I Built a Personal Learning Pipeline Where AI Writes Me a Study Article Every Day

I'm a Japanese software engineer working in the Bay Area. After several years in payments at PayPal, I moved to Atlas, a fintech startup, where I still spend my days close to money movement, ledgers, and all the fun failure modes that come with them. Like a lot of engineers at this stage of their career, I have a long mental list of "things I should really understand properly" — consistent hashing, isolation levels, L4 vs L7 load balancing, idempotency in payment retries — and, like a lot of engineers, that list mostly just grew.

The problem was never motivation. It was consistency and friction. Deciding what to study each day is itself a chore, and by the time I'd picked something and found a good resource, my commute was over. So I built a system that removes both decisions: every morning, a Cloud Function picks a topic from my backlog, has an LLM write a structured study article about it, saves it to Firestore, and publishes it to the study section of my portfolio site. When I open my phone on the train, there's a fresh article waiting — and, downstream, an auto-generated quiz and even a podcast version of it (that's a separate post).

This post walks through how the pipeline works, the parts that turned out to be surprisingly hard (topic repetition, structured output validation), what it costs, and what I'd do differently.

The architecture at a glance

The whole thing runs on my usual stack: Firebase Functions (TypeScript) for compute, Firestore for storage, and a Next.js frontend for reading. Here's the shape of it:

                     (hourly cron, UTC)
 Cloud Scheduler ──▶ dailyArticleGeneration()
                           │
                           ▼
                  study_schedules (Firestore)
                  "should this schedule run at this hour?"
                           │ yes
                           ▼
                   ┌─── topic selection ────┐
                   │ sequential / random    │
                   │ ai_suggested           │
                   │ linear ◀── Linear API  │  (label: Study, state: Todo)
                   └───────────┬────────────┘
                               ▼
                   LLM → structured JSON article
                   (title / summary / sections /
                    key takeaways / quiz questions)
                               │
                               ▼
                    study_articles (Firestore)
                               │
             ┌─────────────────┼───────────────────┐
             ▼                 ▼                   ▼
      Next.js reading UI    quiz docs       Firestore trigger
      (view counts, notes,                  → podcast audio
       article chat)                          generation

A few design decisions worth calling out before diving into each stage.

The scheduler is dumb on purpose. Instead of one Cloud Scheduler job per schedule, a single function runs every hour and scans a study_schedules collection for schedules that should fire at that hour. Schedules are just Firestore documents — frequency, times (stored in UTC), AI provider and model, output language, topic selection mode. Adding or pausing a schedule is a document write, not an infrastructure change.

Everything downstream keys off the article document. Once an article lands in study_articles, other features hang off it without the generator knowing: the reading UI increments a view count, a quiz document links back to it, an article-scoped chat lets me ask follow-up questions with the article as context, and a Firestore onDocumentCreated trigger kicks off text-to-speech podcast generation. The pipeline stays a pipeline; features attach at the edges.

Scheduling: an hourly scan with idempotency guards

The scheduled entry point looks roughly like this (simplified from the real code, as with all snippets in this post):

export const dailyArticleGeneration = onSchedule(
  { schedule: "0 * * * *", timeZone: "UTC", timeoutSeconds: 540, memory: "1GiB" },
  async () => {
    const currentHour = new Date().getUTCHours().toString().padStart(2, "0");

    const schedules = await db
      .collection("study_schedules")
      .where("status", "==", "active")
      .get();

    for (const doc of schedules.docs) {
      if (shouldRunNow(doc.data() as Schedule, currentHour)) {
        await executeSchedule(doc.id);
      }
    }
  }
);

function shouldRunNow(schedule: Schedule, currentHour: string): boolean {
  // Does any scheduled time match this hour?
  const matches = schedule.scheduledTimes.some(
    (t) => t.split(":")[0] === currentHour
  );
  if (!matches) return false;

  // Idempotency guard: skip if we already ran at this hour today (UTC).
  if (schedule.lastRunAt) {
    const last = schedule.lastRunAt.toDate();
    const sameUtcDay =
      last.toISOString().slice(0, 10) === new Date().toISOString().slice(0, 10);
    if (sameUtcDay && last.getUTCHours() === Number(currentHour)) return false;
  }
  return true;
}

The lastRunAt guard matters more than it looks. Scheduled functions can occasionally be invoked more than once, and without the guard you get two articles (and two LLM bills) for the same slot. Each run also writes a schedule_runs record — status, selected topic IDs, produced article IDs, error message if any — which is basically free observability when something silently stops working.

Topic selection: turning a Linear backlog into a curriculum

This is my favorite part of the system. The schedule document has a topicSelectionMode with four strategies:

• sequential — rotate through an explicit topic list, persisting a cursor (nextTopicIndex) so the rotation survives across runs.
• random — shuffle active topics in the selected categories.
• ai_suggested — ask the LLM to propose a new topic, given the category and everything I've already covered.
• linear — pull the oldest open issue from my Linear workspace.

The Linear mode is the one I actually live in. I already capture "I want to understand X" thoughts as Linear issues — it takes five seconds from my phone. So the pipeline queries Linear's GraphQL API for issues with the Study label in the Todo state, oldest first, and eats the backlog from the top:

const issues = await listOpenStudyIssues(); // team YUD, label "Study", state Todo

const issue = issues[0]; // oldest first — FIFO through the backlog
const topicName = topicNameFromIssue(issue);

// The issue body becomes a curriculum outline for the article.
const customPrompt = issue.description
  ? `This article must cover the topics listed in the issue body below.
     Use it as a curriculum outline, but flesh out each point at
     working senior-engineer depth.\n---\n${issue.description}\n---`
  : undefined;

// ...generate and save the article...

// Close the loop: move the issue to In Progress and drop a comment
// containing the published article URL.
await markIssueArticleGenerated(issue, articleUrl);

Two details I'd highlight. First, the issue body isn't ignored — if I jot bullet points under the title ("cover leader election, fencing tokens, lease expiry"), those become the article's outline. A vague thought at capture time turns into a structured syllabus at generation time. Second, the loop is closed: the issue is moved to In Progress and gets a comment linking to the generated article, so my Linear board doubles as a "what have I studied" log. Each Linear issue is also mapped to a Firestore topic document via a linear:{identifier} tag, so regenerating an article for the same issue reuses the same topic instead of creating duplicates.

The repetition problem (harder than expected)

If you ask an LLM to "suggest a system design topic" every day, you will learn a lot about caching. And then more about caching. My generator developed pet topics — caching, Kubernetes deployments, and for some reason an intense obsession with the Outbox pattern and Debezium — and it took several layers of defense to get real diversity:

1. Recent titles in the prompt. Every generation fetches up to 500 recent article titles from Firestore and includes the newest 50 in the prompt as "avoid duplicates/similar phrasing."

2. Theme frequency counting. Titles are matched against regex-defined themes; if a theme appears three or more times in the recent window, the prompt gets an explicit "recently overused themes — do not pick these again right now" section:

const THEMES = [
  { id: "cache", label: "Caching / Redis / CDN",
    patterns: [/\bcache\b/i, /redis/i, /\bcdn\b/i, /キャッシュ/] },
  { id: "kubernetes", label: "Kubernetes / deployments",
    patterns: [/kubernetes/i, /\bk8s\b/i] },
];

const counts = countThemes(recentTitles.slice(0, 200), THEMES);
const overused = THEMES.filter((t) => counts[t.id] >= 3);
// → prompt gains: "RECENTLY OVERUSED THEMES (avoid):
//    - Caching / Redis / CDN (5 recent titles)"

3. Hard forbidden-topic filters. Some topics were so overproduced that I banned them outright with regex patterns checked both at topic selection and at title validation. Yes, my codebase contains a constant that effectively says "no more Debezium articles." I'm at peace with this.

4. Diversity buckets within a run. When one run generates multiple articles, each candidate topic is classified into a bucket (caching, databases, messaging, security, case studies...) and a bucket can't be used twice in the same run.

5. Post-generation title validation with retry. Even with all of the above in the prompt, the model sometimes produces a near-duplicate title anyway. So the generated title is validated after the fact, and rejected generations are retried with the rejection reason fed back:

for (let attempt = 1; attempt <= 3; attempt++) {
  const raw = await callLLM(buildPrompt({ topic, recentTitles, retryNote }));
  const article = parseArticleJson(raw);

  const check = validateTitle(article.title, recentTitles);
  if (check.ok) return article;

  retryNote =
    `The previous title "${article.title}" was rejected ` +
    `(${check.reasons.join(", ")}). Pick a clearly different angle.`;
}
throw new Error("Title failed validation after 3 attempts");

The general lesson: prompt-level "please avoid X" is a suggestion; code-level validation with retry is a guarantee. You want both, because validation-only means wasted API calls and prompt-only means duplicates slip through.

On difficulty progression: each topic and article carries a difficulty level (beginner → expert), and the prompt maps each level to a concrete reader description — intermediate is "a senior engineer deepening system-design judgment: emphasize trade-offs, decision criteria, failure modes," advanced adds scaling cliffs and second-order effects. In practice I keep the daily schedule at intermediate and bump specific topics up when a first-pass article leaves me wanting more; a fully automatic "you've read three articles on this theme, escalating difficulty" ladder is on the future-ideas list.

Structured output and validation

Articles aren't markdown blobs. The LLM must return a JSON object matching a typed contract, because the reading UI renders sections, code blocks, and quizzes as distinct components:

interface GeneratedArticle {
  title: string;
  summary: string;
  introduction: string;
  sections: {
    title: string;
    content: string;             // markdown
    codeExamples?: { language: string; code: string; explanation: string }[];
    externalLinks?: { title: string; url: string; description: string }[];
  }[];
  conclusion: string;
  keyTakeaways: string[];
  suggestedQuestions?: QuizQuestion[];  // becomes a quiz document
}

Parsing the response is defensive by necessity:

function parseArticleJson(responseText: string): GeneratedArticle {
  // Models love wrapping JSON in ```json fences — and the article itself
  // contains ``` fences inside code examples, so find the OUTERMOST block.
  const jsonStr = extractOutermostJsonBlock(responseText);

  // Long articles sometimes hit the output-token ceiling mid-object.
  if (!jsonStr.trimEnd().endsWith("}")) {
    throw new Error("AI response was truncated - JSON incomplete");
  }

  const parsed = JSON.parse(jsonStr);
  if (!parsed.title || !parsed.summary || !parsed.sections) {
    throw new Error("Parsed JSON missing required fields");
  }
  return parsed as GeneratedArticle;
}

Every failure mode in that function is one I actually hit: fenced output, truncated output on long articles, valid JSON missing required fields. Before writing to Firestore, undefined values are stripped field-by-field (Firestore rejects them — a classic paper cut), a reading time is computed at 100 words per minute (technical content reads slower than prose), and a URL slug is derived from the title.

There's also a dry-run mode: pass dryRun: true to the generation endpoint and it resolves the topic, builds the exact prompt, and returns it with debug info (how many existing titles were included, theme counts, prompt length) — without calling the LLM or writing anything. When generation quality drifts, being able to inspect precisely what the model was told, for free, is the single most useful debugging tool in the system.

Prompt versioning and prompt design

The system prompt is the product here, and it has been through many revisions. The current version encodes hard-won rules: write for a working senior engineer, don't explain what a cache is; explain the mechanism cleanly before the trade-offs; include at least one trade-off comparison table; name real production failure modes (hot keys, thundering herd, retry amplification) rather than inventing exotic ones; and — my favorite rule, added after one restaurant analogy too many — absolutely no everyday-life analogies. If a concept needs grounding, ground it in a request-response cycle, a database write, a payment flow.

Because the prompt keeps evolving, versioning matters. Config lives in a Firestore document (provider, model names, temperature, defaults), so switching models is a document edit. Each article records which provider and model produced it, and the prompt itself carries a version number — which becomes genuinely important downstream, where the podcast pipeline uses the prompt version as a cache-busting input (more on that in the companion post). Language is a first-class parameter too: I mostly generate in Japanese, and the prompt includes explicit Japanese style instructions (consistent です・ます register, sensible katakana-vs-English term choices, code and comments stay in English).

What it costs

Less than you'd guess. A daily article is one large LLM call (a few thousand input tokens, a long structured output), plus a cheap topic-suggestion call in ai_suggested mode and up to two retries on validation failure — in practice retries fire maybe one day in five. With a mid-tier model this lands around $0.05–0.15 per article, so roughly $2–5 per month for a daily habit. Firebase costs are noise: Firestore reads/writes and the hourly scheduler invocations sit comfortably in the free tier, and the functions are 1GiB instances that run for tens of seconds a day. Compared to one technical book a month, it's not even a conversation.

What worked, what didn't

Worked:

• Removing the daily decision. The article is just there every morning. Zero-friction consistency beat every studying app I've tried, because the failure mode ("I didn't pick anything today") is gone.
• Linear as the topic source. Capture is five seconds, and the closed loop (issue → article → comment with URL → In Progress) makes the backlog feel alive instead of guilt-inducing.
• Issue bodies as curricula. Bullet points written at capture time steer the article's structure surprisingly well.
• Structured output. Because articles are typed data rather than markdown blobs, quizzes, audio, chat, and flashcards attached later without touching the generator.

Didn't work (initially or still):

• Naive topic suggestion. The pet-topic problem was real and needed five layers of defense. Title-based dedupe is still shallow — two articles can share no title words and still cover 70% of the same material. Embedding-based similarity over article bodies is the proper fix.
• JSON truncation. Long articles hitting output-token limits was a recurring, annoying failure until detection (and prompt-side length discipline) improved.
• Passive reading. An article every day doesn't mean retention every day. Quizzes help; the honest fix is spaced repetition driven by quiz results, which is only partly built.
• Volume management. For a while I generated multiple articles a day, which just moved the guilt from the Linear backlog to an unread-articles backlog. One good article a day is the sustainable setting.

Future ideas: semantic dedupe with embeddings; automatic difficulty ladders (regenerate a topic one level up after a passed quiz); feeding quiz mistakes back into the next generation ("the reader struggled with fencing tokens — revisit from a different angle"); and weekly digest articles synthesizing the week's topics into one review piece.

Summary

• A single hourly Cloud Function scanning schedule documents in Firestore is a perfectly good scheduler — just add an idempotency guard (lastRunAt hour check) and a run-log collection.
• Sourcing topics from a Linear label turns "I should learn this someday" into "this is tomorrow's reading," and closing the loop (In Progress + comment with the article URL) makes the backlog self-documenting.
• LLMs have pet topics. Prompt-level avoidance (recent titles, overused-theme counts) reduces repetition; code-level validation with retry-and-feedback prevents it. Use both.
• Demand structured JSON, then parse it defensively: outermost-fence extraction, truncation detection, required-field checks, undefined stripping for Firestore.
• A dry-run mode that returns the exact prompt without calling the model is the cheapest, highest-value debugging feature you can add to any LLM pipeline.
• The whole thing costs a few dollars a month — the constraint on learning infrastructure is design effort, not budget.

The meta-lesson: the highest-leverage thing I automated wasn't the writing — it was the deciding. If you're an engineer with a backlog of things you want to learn, you already have the topic pipeline. It's sitting in your issue tracker.