24 KiB

Raw Blame History

User Data Export Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Build a user-owned export pipeline that produces a versioned manifest + data JSON + files package with configurable masking and secure download.

Architecture: Add an async export job table and service pipeline in Nitro server: request API creates job, worker executes export with a fixed cutoff timestamp, and download API serves completed artifacts. Export logic is split into small modules for domain data extraction, masking/sanitization, file collection, checksum generation, and manifest assembly. UI exposes a simple “Data Export Center” under /me for request, progress, and download.

Tech Stack: Nuxt 4, Nitro server routes/tasks, Bun test, Drizzle ORM (sqlite schema + migrations), Node fs/crypto/path APIs, Vue + Nuxt UI.

File Structure (planned changes)

Create: packages/drizzle-pkg/database/sqlite/schema/export.ts（导出任务表）
Modify: packages/drizzle-pkg/database/sqlite/schema/index.ts（导出新 schema）
Modify: packages/drizzle-pkg/lib/schema/content.ts（按现有导出方式补 export schema）
Create: packages/drizzle-pkg/migrations/0010_user_export_tasks.sql
Create: server/constants/export.ts（状态、策略、路径、TTL）
Create: server/utils/export-mask.ts（脱敏与永不导出过滤）
Create: server/utils/export-hash.ts（SHA256 计算）
Create: server/service/export/jobs.ts（任务创建/查询/状态机）
Create: server/service/export/build-data.ts（业务数据抽取）
Create: server/service/export/build-files.ts（媒体文件收集）
Create: server/service/export/build-manifest.ts（manifest 组装）
Create: server/service/export/run.ts（任务执行主流程）
Create: server/tasks/export/process.ts（定时/触发执行任务）
Create: server/api/me/export/request.post.ts
Create: server/api/me/export/tasks.get.ts
Create: server/api/me/export/tasks/[id]/download.get.ts
Create: server/utils/me-export-request-body.ts（请求参数校验）
Create: server/utils/me-export-request-body.test.ts
Create: server/service/export/jobs.test.ts
Create: server/service/export/build-manifest.test.ts
Modify: app/pages/me/index.vue（加入“数据导出中心”入口）
Create: app/pages/me/export/index.vue（导出页面）
Create: app/types/export.ts（前端 DTO）

Task 1: Export Job Schema + Migration

Files:

Create: packages/drizzle-pkg/database/sqlite/schema/export.ts
Modify: packages/drizzle-pkg/database/sqlite/schema/index.ts
Modify: packages/drizzle-pkg/lib/schema/content.ts
Create: packages/drizzle-pkg/migrations/0010_user_export_tasks.sql
Test: bun run db:migrate
Step 1: Write the failing schema export test (smoke)

// File: server/service/export/jobs.test.ts (temporary first test)
import { describe, expect, test } from "bun:test";
import { userExportTasks } from "drizzle-pkg/lib/schema/export";

describe("userExportTasks schema", () => {
  test("exposes required columns", () => {
    expect(userExportTasks.userId.name).toBe("user_id");
    expect(userExportTasks.status.name).toBe("status");
    expect(userExportTasks.maskPolicy.name).toBe("mask_policy");
  });
});

Step 2: Run test to verify it fails

Run: bun test server/service/export/jobs.test.ts
Expected: FAIL with module or symbol not found for drizzle-pkg/lib/schema/export.

Step 3: Add minimal schema + migration

// File: packages/drizzle-pkg/database/sqlite/schema/export.ts
import { index, integer, sqliteTable, text } from "drizzle-orm/sqlite-core";
import { users } from "./auth";

export const userExportTasks = sqliteTable(
  "user_export_tasks",
  {
    id: integer().primaryKey(),
    userId: integer("user_id").notNull().references(() => users.id, { onDelete: "cascade" }),
    status: text().notNull().default("queued"), // queued|running|succeeded|failed|expired
    maskPolicy: text("mask_policy").notNull().default("masked"), // masked|raw
    exportCutoffAt: integer("export_cutoff_at", { mode: "timestamp_ms" }),
    outputDir: text("output_dir"),
    outputName: text("output_name"),
    totalBytes: integer("total_bytes"),
    errorCode: text("error_code"),
    errorMessage: text("error_message"),
    expiresAt: integer("expires_at", { mode: "timestamp_ms" }),
    createdAt: integer("created_at", { mode: "timestamp_ms" }).defaultNow().notNull(),
    updatedAt: integer("updated_at", { mode: "timestamp_ms" }).defaultNow().$onUpdate(() => new Date()).notNull(),
  },
  (table) => [
    index("user_export_tasks_user_id_idx").on(table.userId),
    index("user_export_tasks_status_idx").on(table.status),
  ],
);

-- File: packages/drizzle-pkg/migrations/0010_user_export_tasks.sql
CREATE TABLE user_export_tasks (
  id INTEGER PRIMARY KEY NOT NULL,
  user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
  status TEXT NOT NULL DEFAULT 'queued',
  mask_policy TEXT NOT NULL DEFAULT 'masked',
  export_cutoff_at INTEGER,
  output_dir TEXT,
  output_name TEXT,
  total_bytes INTEGER,
  error_code TEXT,
  error_message TEXT,
  expires_at INTEGER,
  created_at INTEGER NOT NULL DEFAULT (unixepoch('subsec') * 1000),
  updated_at INTEGER NOT NULL DEFAULT (unixepoch('subsec') * 1000)
);
CREATE INDEX user_export_tasks_user_id_idx ON user_export_tasks(user_id);
CREATE INDEX user_export_tasks_status_idx ON user_export_tasks(status);

Step 4: Run tests and migration

Run: bun test server/service/export/jobs.test.ts && bun run db:migrate
Expected: PASS test; migration applies without SQL error.

Step 5: Commit

git add packages/drizzle-pkg/database/sqlite/schema/export.ts \
  packages/drizzle-pkg/database/sqlite/schema/index.ts \
  packages/drizzle-pkg/lib/schema/content.ts \
  packages/drizzle-pkg/migrations/0010_user_export_tasks.sql \
  server/service/export/jobs.test.ts
git commit -m "feat(export): add user export task schema and migration"

Task 2: Request Validation + Masking Rules

Files:

Create: server/constants/export.ts
Create: server/utils/me-export-request-body.ts
Create: server/utils/me-export-request-body.test.ts
Create: server/utils/export-mask.ts
Test: server/utils/me-export-request-body.test.ts
Step 1: Write failing tests for request body and masking behavior

// File: server/utils/me-export-request-body.test.ts
import { describe, expect, test } from "bun:test";
import { parseMeExportRequestBody } from "./me-export-request-body";
import { sanitizeUserForExport } from "./export-mask";

describe("parseMeExportRequestBody", () => {
  test("defaults to masked when absent", () => {
    expect(parseMeExportRequestBody({})).toEqual({ maskPolicy: "masked" });
  });
  test("rejects invalid policy", () => {
    expect(() => parseMeExportRequestBody({ maskPolicy: "abc" })).toThrow();
  });
});

describe("sanitizeUserForExport", () => {
  test("never exports password/session-like fields", () => {
    const sanitized = sanitizeUserForExport({ username: "u", password: "hash", sessionToken: "x" } as any, "raw");
    expect("password" in sanitized).toBe(false);
    expect("sessionToken" in sanitized).toBe(false);
  });
});

Step 2: Run test to verify it fails

Run: bun test server/utils/me-export-request-body.test.ts
Expected: FAIL with missing modules/functions.

Step 3: Implement parser + masking utility

// File: server/utils/me-export-request-body.ts
export type ExportMaskPolicy = "masked" | "raw";

export function parseMeExportRequestBody(body: unknown): { maskPolicy: ExportMaskPolicy } {
  const payload = (body && typeof body === "object" ? body : {}) as { maskPolicy?: unknown };
  const raw = payload.maskPolicy;
  if (raw === undefined || raw === null || raw === "") return { maskPolicy: "masked" };
  if (raw === "masked" || raw === "raw") return { maskPolicy: raw };
  throw createError({ statusCode: 400, statusMessage: "maskPolicy 仅支持 masked/raw" });
}

// File: server/utils/export-mask.ts
const NEVER_EXPORT_KEYS = new Set([
  "password",
  "passwordHash",
  "sessionToken",
  "refreshToken",
  "resetToken",
  "apiKey",
  "secretKey",
]);

export function sanitizeUserForExport<T extends Record<string, unknown>>(input: T, policy: "masked" | "raw") {
  const out: Record<string, unknown> = {};
  for (const [k, v] of Object.entries(input)) {
    if (NEVER_EXPORT_KEYS.has(k)) continue;
    out[k] = policy === "masked" ? maskField(k, v) : v;
  }
  return out;
}

function maskField(key: string, value: unknown): unknown {
  if (typeof value !== "string") return value;
  if (key.includes("email")) return value.replace(/^(.).+(@.+)$/, "$1***$2");
  if (key.includes("phone")) return value.replace(/^(\d{3})\d+(\d{2})$/, "$1****$2");
  if (key.toLowerCase().includes("ip")) return value.replace(/^(\d+\.\d+)\.\d+\.\d+$/, "$1.*.*");
  return value;
}

Step 4: Run tests to verify pass

Run: bun test server/utils/me-export-request-body.test.ts
Expected: PASS.

Step 5: Commit

git add server/constants/export.ts \
  server/utils/me-export-request-body.ts \
  server/utils/me-export-request-body.test.ts \
  server/utils/export-mask.ts
git commit -m "feat(export): add request validation and masking utilities"

Task 3: Export Job Service (create/list/update)

Files:

Create: server/service/export/jobs.ts
Modify: server/service/export/jobs.test.ts
Test: server/service/export/jobs.test.ts
Step 1: Write failing tests for job lifecycle

// File: server/service/export/jobs.test.ts
import { describe, expect, test } from "bun:test";
import { createExportTask, listExportTasksByUser, markExportTaskRunning } from "./jobs";

describe("export jobs service", () => {
  test("createExportTask creates queued task", async () => {
    const task = await createExportTask({ userId: 1, maskPolicy: "masked" });
    expect(task.status).toBe("queued");
    expect(task.maskPolicy).toBe("masked");
  });

  test("markExportTaskRunning sets cutoff timestamp", async () => {
    const task = await createExportTask({ userId: 1, maskPolicy: "raw" });
    const updated = await markExportTaskRunning(task.id);
    expect(updated.status).toBe("running");
    expect(updated.exportCutoffAt).toBeTruthy();
  });

  test("listExportTasksByUser only returns own tasks", async () => {
    const rows = await listExportTasksByUser(1);
    expect(Array.isArray(rows)).toBe(true);
  });
});

Step 2: Run test to verify it fails

Run: bun test server/service/export/jobs.test.ts
Expected: FAIL with missing implementation.

Step 3: Implement minimal job service

// File: server/service/export/jobs.ts
import { dbGlobal } from "drizzle-pkg/lib/db";
import { userExportTasks } from "drizzle-pkg/lib/schema/export";
import { and, desc, eq } from "drizzle-orm";
import { nextIntegerId } from "#server/utils/sqlite-id";

export async function createExportTask(params: { userId: number; maskPolicy: "masked" | "raw" }) {
  const id = await nextIntegerId(userExportTasks, userExportTasks.id);
  await dbGlobal.insert(userExportTasks).values({
    id,
    userId: params.userId,
    maskPolicy: params.maskPolicy,
    status: "queued",
  });
  const [row] = await dbGlobal.select().from(userExportTasks).where(eq(userExportTasks.id, id)).limit(1);
  return row!;
}

export async function listExportTasksByUser(userId: number) {
  return dbGlobal.select().from(userExportTasks).where(eq(userExportTasks.userId, userId)).orderBy(desc(userExportTasks.id));
}

export async function markExportTaskRunning(taskId: number) {
  const cutoff = new Date();
  await dbGlobal.update(userExportTasks).set({ status: "running", exportCutoffAt: cutoff }).where(eq(userExportTasks.id, taskId));
  const [row] = await dbGlobal.select().from(userExportTasks).where(eq(userExportTasks.id, taskId)).limit(1);
  return row!;
}

export async function markExportTaskSucceeded(taskId: number, payload: { outputDir: string; outputName: string; totalBytes: number; expiresAt: Date }) {
  await dbGlobal
    .update(userExportTasks)
    .set({ status: "succeeded", ...payload })
    .where(eq(userExportTasks.id, taskId));
}

export async function markExportTaskFailed(taskId: number, payload: { errorCode: string; errorMessage: string }) {
  await dbGlobal.update(userExportTasks).set({ status: "failed", ...payload }).where(eq(userExportTasks.id, taskId));
}

export async function getExportTaskForUser(taskId: number, userId: number) {
  const [row] = await dbGlobal
    .select()
    .from(userExportTasks)
    .where(and(eq(userExportTasks.id, taskId), eq(userExportTasks.userId, userId)))
    .limit(1);
  return row ?? null;
}

Step 4: Run tests to verify pass

Run: bun test server/service/export/jobs.test.ts
Expected: PASS.

Step 5: Commit

git add server/service/export/jobs.ts server/service/export/jobs.test.ts
git commit -m "feat(export): add export task lifecycle service"

Task 4: Build Export Artifacts (data/files/manifest)

Files:

Create: server/service/export/build-data.ts
Create: server/service/export/build-files.ts
Create: server/service/export/build-manifest.ts
Create: server/service/export/build-manifest.test.ts
Create: server/utils/export-hash.ts
Test: server/service/export/build-manifest.test.ts
Step 1: Write failing test for manifest shape + checksum keys

// File: server/service/export/build-manifest.test.ts
import { describe, expect, test } from "bun:test";
import { buildExportManifest } from "./build-manifest";

describe("buildExportManifest", () => {
  test("outputs schemaVersion and checksums", () => {
    const manifest = buildExportManifest({
      schemaVersion: 1,
      userId: 1,
      maskPolicy: "masked",
      exportCutoffAt: new Date("2026-04-24T00:00:00.000Z"),
      dataChecksums: [{ file: "data/user.json", sha256: "a" }],
      fileChecksums: [{ file: "files/a.webp", sha256: "b" }],
      stats: { dataRows: 10, files: 1, bytes: 1234 },
    });
    expect(manifest.schemaVersion).toBe(1);
    expect(manifest.checksums.files[0].file).toBe("files/a.webp");
  });
});

Step 2: Run test to verify fail

Run: bun test server/service/export/build-manifest.test.ts
Expected: FAIL with missing module/function.

Step 3: Implement data/files/manifest builders

// File: server/service/export/build-data.ts (core signature)
export async function buildExportDataJson(params: {
  userId: number;
  cutoffAt: Date;
  maskPolicy: "masked" | "raw";
  outputDataDir: string;
}): Promise<{ files: Array<{ file: string; rowCount: number }> }> {
  // 1) query users/profile/posts/timeline/comments/mediaAssets/mediaRefs up to cutoffAt
  // 2) sanitize with export-mask policy
  // 3) write JSON files into outputDataDir
  // 4) return file list + row counts
}

// File: server/service/export/build-files.ts (core signature)
export async function buildExportMediaFiles(params: {
  userId: number;
  cutoffAt: Date;
  outputFilesDir: string;
}): Promise<{ files: Array<{ file: string; bytes: number }> }> {
  // query media_assets by user, copy each storageKey into files/, keep deterministic relative paths
}

// File: server/service/export/build-manifest.ts
export function buildExportManifest(input: {
  schemaVersion: number;
  userId: number;
  maskPolicy: "masked" | "raw";
  exportCutoffAt: Date;
  dataChecksums: Array<{ file: string; sha256: string }>;
  fileChecksums: Array<{ file: string; sha256: string }>;
  stats: { dataRows: number; files: number; bytes: number };
}) {
  return {
    schemaVersion: input.schemaVersion,
    exportedAt: new Date().toISOString(),
    exportCutoffAt: input.exportCutoffAt.toISOString(),
    userId: input.userId,
    maskPolicy: input.maskPolicy,
    stats: input.stats,
    checksums: { data: input.dataChecksums, files: input.fileChecksums },
  };
}

Step 4: Run tests to verify pass

Run: bun test server/service/export/build-manifest.test.ts
Expected: PASS.

Step 5: Commit

git add server/service/export/build-data.ts \
  server/service/export/build-files.ts \
  server/service/export/build-manifest.ts \
  server/service/export/build-manifest.test.ts \
  server/utils/export-hash.ts
git commit -m "feat(export): add export artifact builders and manifest"

Task 5: Runner + Task Processor + Me APIs

Files:

Create: server/service/export/run.ts
Create: server/tasks/export/process.ts
Create: server/api/me/export/request.post.ts
Create: server/api/me/export/tasks.get.ts
Create: server/api/me/export/tasks/[id]/download.get.ts
Modify: server/service/export/jobs.ts（补 claimNextQueuedTask）
Test: server/service/export/jobs.test.ts + manual API smoke
Step 1: Write failing tests for queue claim behavior

// File: server/service/export/jobs.test.ts (append)
import { claimNextQueuedTask } from "./jobs";

test("claimNextQueuedTask returns null when no queued task", async () => {
  const task = await claimNextQueuedTask();
  expect(task).toBeNull();
});

Step 2: Run test to verify fail

Run: bun test server/service/export/jobs.test.ts
Expected: FAIL with missing function.

Step 3: Implement runner and APIs

// File: server/service/export/run.ts
export async function runExportTask(taskId: number): Promise<void> {
  const running = await markExportTaskRunning(taskId);
  try {
    // create output dirs
    // build data json + files
    // hash files + build manifest + write manifest.json
    // mark succeeded with expiresAt
  } catch (error) {
    await markExportTaskFailed(taskId, {
      errorCode: "EXPORT_BUILD_FAILED",
      errorMessage: error instanceof Error ? error.message : "unknown export error",
    });
  }
}

// File: server/api/me/export/request.post.ts
export default defineWrappedResponseHandler(async (event) => {
  const user = await event.context.auth.requireUser();
  const body = await readBody(event);
  const { maskPolicy } = parseMeExportRequestBody(body);
  const task = await createExportTask({ userId: user.id, maskPolicy });
  return R.success({ taskId: task.id, status: task.status });
});

// File: server/api/me/export/tasks.get.ts
export default defineWrappedResponseHandler(async (event) => {
  const user = await event.context.auth.requireUser();
  const tasks = await listExportTasksByUser(user.id);
  return R.success({
    items: tasks.map((t) => ({
      id: t.id,
      status: t.status,
      maskPolicy: t.maskPolicy,
      createdAt: t.createdAt.toISOString(),
      expiresAt: t.expiresAt?.toISOString() ?? null,
    })),
  });
});

// File: server/api/me/export/tasks/[id]/download.get.ts
export default defineWrappedResponseHandler(async (event) => {
  const user = await event.context.auth.requireUser();
  const id = Number(getRouterParam(event, "id"));
  const task = await getExportTaskForUser(id, user.id);
  if (!task || task.status !== "succeeded" || !task.outputDir) {
    throw createError({ statusCode: 404, statusMessage: "导出文件不存在或未完成" });
  }
  // serve file/dir stream according to deployment convention
});

Step 4: Run tests and smoke endpoints

Run:

bun test server/service/export/jobs.test.ts
bun run dev then call:
- POST /api/me/export/request
- GET /api/me/export/tasks

Expected:

tests PASS
request returns queued task id
list endpoint contains new task.
Step 5: Commit

git add server/service/export/run.ts \
  server/tasks/export/process.ts \
  server/api/me/export/request.post.ts \
  server/api/me/export/tasks.get.ts \
  server/api/me/export/tasks/[id]/download.get.ts \
  server/service/export/jobs.ts \
  server/service/export/jobs.test.ts
git commit -m "feat(export): add export runner, processor task, and me export APIs"

Task 6: Frontend Export Center (`/me/export`)

Files:

Modify: app/pages/me/index.vue
Create: app/pages/me/export/index.vue
Create: app/types/export.ts
Test: manual browser verification
Step 1: Write lightweight UI contract notes as failing checklist

// File: app/pages/me/export/index.vue (top TODO block during implementation)
- must show "发起导出" button
- must allow selecting mask policy (masked/raw)
- must show task list status + created time
- must show download button only when succeeded

Step 2: Run app and verify page not present yet (expected fail)

Run: bun run dev and open /me/export
Expected: 404 or missing route behavior.

Step 3: Implement page and dashboard entry

<!-- File: app/pages/me/export/index.vue -->
<script setup lang="ts">
const { fetchData } = useClientApi();
const maskPolicy = ref<"masked" | "raw">("masked");
const loading = ref(false);
const tasks = ref<Array<{ id: number; status: string; maskPolicy: string; createdAt: string; expiresAt: string | null }>>([]);

async function loadTasks() {
  const res = await fetchData<{ items: typeof tasks.value }>("/api/me/export/tasks");
  tasks.value = res.items;
}

async function createTask() {
  loading.value = true;
  try {
    await fetchData("/api/me/export/request", { method: "POST", body: { maskPolicy: maskPolicy.value } });
    await loadTasks();
  } finally {
    loading.value = false;
  }
}

onMounted(loadTasks);
</script>

<!-- File: app/pages/me/index.vue (new card) -->
<UCard>
  <div class="font-medium">数据导出</div>
  <p class="text-sm text-muted mt-1">导出用户文件与业务数据用于迁移。</p>
  <UButton to="/me/export" class="mt-3" size="sm">进入</UButton>
</UCard>

Step 4: Manual verify pass

Run: bun run dev
Expected:

/me/export 可访问
可创建导出任务
成功任务显示下载入口。
Step 5: Commit

git add app/pages/me/index.vue app/pages/me/export/index.vue app/types/export.ts
git commit -m "feat(export): add me export center page and task actions"

Task 7: Security/Regression Tests + Docs

Files:

Modify: server/service/export/jobs.test.ts
Modify: server/utils/me-export-request-body.test.ts
Modify: README.md
Modify: docs/superpowers/specs/2026-04-24-user-data-backup-design.md（实现状态备注）
Test: targeted bun tests + lint
Step 1: Add failing security regression tests

// File: server/utils/me-export-request-body.test.ts (append)
test("raw policy still excludes never-export fields", () => {
  const data = sanitizeUserForExport({ password: "x", apiKey: "k", email: "a@b.com" } as any, "raw");
  expect(data).toEqual({ email: "a@b.com" });
});

Step 2: Run tests to verify failure (if behavior regresses)

Run: bun test server/utils/me-export-request-body.test.ts
Expected: PASS only when blacklist behavior is correct.

Step 3: Update docs with usage commands

<!-- File: README.md (append section) -->
## 用户数据导出（迁移）

- 发起导出：`POST /api/me/export/request`，body: `{ "maskPolicy": "masked" | "raw" }`
- 查询任务：`GET /api/me/export/tasks`
- 下载结果：`GET /api/me/export/tasks/:id/download`

Step 4: Final verification

Run:

bun test server/service/export/jobs.test.ts
bun test server/service/export/build-manifest.test.ts
bun test server/utils/me-export-request-body.test.ts

Expected: all PASS.

Step 5: Commit

git add server/service/export/jobs.test.ts \
  server/utils/me-export-request-body.test.ts \
  README.md \
  docs/superpowers/specs/2026-04-24-user-data-backup-design.md
git commit -m "test/docs(export): add security regression coverage and usage docs"

Self-Review Checklist

Spec coverage:
- 异步任务模型：Task 1, 3, 5
- 目录+清单导出：Task 4
- 全量用户域数据 + 文件：Task 4
- 脱敏与永不导出：Task 2, 7
- 安全下载与过期：Task 5
- 前端入口与操作：Task 6
Placeholder scan: 已避免 TODO/TBD；每个任务给出代码/命令/预期。
Type consistency:
- maskPolicy 统一 masked | raw
- 状态统一 queued|running|succeeded|failed|expired
- 清单版本固定 schemaVersion: 1

24 KiB Raw Blame History

User Data Export Implementation Plan

File Structure (planned changes)

Task 1: Export Job Schema + Migration

Task 2: Request Validation + Masking Rules

Task 3: Export Job Service (create/list/update)

Task 4: Build Export Artifacts (data/files/manifest)

Task 5: Runner + Task Processor + Me APIs

Task 6: Frontend Export Center (/me/export)

Task 7: Security/Regression Tests + Docs

Self-Review Checklist

24 KiB

Raw Blame History

Task 6: Frontend Export Center (`/me/export`)