Background jobs with service accounts

This guide builds LedgerOps, a fictional finance app whose nightly worker exports one organization's ledger. The worker receives a dedicated credential, resolves to service_account::ledger-exporter, and can perform only LEDGER_EXPORT on ledger::northwind.

Recommended: declare one machine client#

SeedMachineClient reconciles the confidential OAuth client, FGA service-account subject, slow credential hash, organization binding, and initial grants as one ownership-safe unit. The resolver reads the secret from the host's existing secret provider; SqlOS never writes or logs the plaintext value:

options.AuthServer.SeedMachineClient("ledger-exporter", (client, machine) =>
{
    client.Name = "Ledger Exporter";
    client.Description = "Nightly read-only export for Northwind";
    client.Audience = "https://api.example.com/ledger";
    client.AllowedScopes = ["ledger.export"];
 
    machine.OrganizationSlug = "northwind";
    machine.ExpiresAt = DateTime.UtcNow.AddDays(90);
    machine.SecretResolver = () =>
        builder.Configuration["SqlOS:MachineClients:LedgerExporter:Secret"];
    machine.Grant("ledger::northwind", "export_reader", "Nightly ledger export");
});

The secret must contain 43 to 256 characters. Missing material fails startup closed; a restart does not generate a new secret. Changing the value in the deployment secret provider performs an explicit code-owned rotation on reconciliation. Removing the declaration marks the machine client orphaned for operator review instead of silently disabling a job.

Alternatively, provide SecretHashResolver when your deployment pipeline already produces an ASP.NET Core PasswordHasher-compatible hash. Specify exactly one resolver.

Dashboard and admin API workflow#

Open Auth Server > Machine Clients to create the same OAuth/FGA identity interactively. Creation is atomic and displays a generated 64-character secret exactly once. Later list/detail responses contain client ID, audience, scopes, organization, expiry, last use, ownership, readiness, and grant count—but never the hash or secret.

The authenticated admin API uses these routes:

Code-owned identities remain visible and testable in the dashboard, but rotation and grant mutation are blocked because startup would otherwise overwrite an operator's change. Update the declaration or secret provider instead. Dashboard-owned records remain fully operable. The generic FGA Service Accounts view still exposes the subject and grant paths and links back conceptually to the same client ID.

Lower-level provisioning remains available#

For advanced application-owned boundaries, the original helpers remain usable. Provision the matching FGA service account from a trusted administrative command and deliver the raw secret directly to the worker's secret manager:

var secret = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
 
await db.ProvisionServiceAccountSubjectAsync(
    subjectId: "service_account::ledger-exporter",
    displayName: "Ledger Exporter",
    clientId: "ledger-exporter",
    clientSecretHash: crypto.HashPassword(secret),
    organizationId: "northwind",
    cancellationToken: ct);
 
await db.GrantRoleAsync(
    "service_account::ledger-exporter",
    "ledger::northwind",
    "export_reader",
    ct);
await db.SaveChangesAsync(ct);

Exchange the credential with HTTP Basic authentication. The resource must exactly match the seeded audience and every scope must be pre-authorized:

curl -u 'ledger-exporter:YOUR_SECRET' \
  -d 'grant_type=client_credentials' \
  -d 'resource=https://api.example.com/ledger' \
  -d 'scope=ledger.export' \
  https://identity.example.com/sqlos/auth/token

The response contains an access token only—never a refresh token. Its sub is service_account::ledger-exporter; it includes client_id, azp, scope, and token_kind=service, but no human email and no sid session claim. Normal audience validation remains mandatory at the resource server.

Code-owned secret rotation is an explicit cutover: replace the value returned by the configured secret resolver, deliver it to the worker, and restart or reconcile the application. Dashboard-managed OAuth credentials can instead overlap during deployment: create a second credential, move the worker, and then revoke the old credential. Already-issued short-lived access tokens remain valid until expiry; set the service account ExpiresAt to the current time to revoke its FGA-backed service tokens immediately. Rotation, successful issuance, and failed authentication are recorded in the audit log.

Public DCR intentionally cannot create confidential machine clients. Create them through trusted code/configuration and provision credentials through an authenticated administrative workflow.

Goal#

The finished request path has four independent controls:

1. a random credential identifies one worker;
2. the stored hash verifies that credential without retaining the secret;
3. an FGA check limits the identity to one permission and resource;
4. an audit event records the attempted export.

The scheduler never stores a person's refresh token and never impersonates a user.

1. Define the narrow role#

Seed a permission and role for the job. Keep machine roles separate from broad human roles so reviewing a grant explains exactly what the worker can do.

options.Fga.Seed(seed =>
{
    seed.ResourceType("ledger", "Ledger");
    seed.Permission("LEDGER_EXPORT", "Export ledger data", "ledger");
    seed.Role("export_reader", "Ledger Export Reader");
    seed.RolePermission("export_reader", "LEDGER_EXPORT");
});

The target ledger::northwind must already exist as an FGA resource, either through an ISqlOSResourceEntity or explicit resource provisioning.

2. Create the credential once#

Run credential creation from an authenticated admin command—not unconditionally at every startup. Generate at least 256 random bits and write the resulting credential directly to the worker's secret manager.

using System.Security.Cryptography;
using SqlOS.AuthServer.Services;
using SqlOS.Extensions;
 
var clientId = "ledger-exporter";
var secret = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
var secretHash = crypto.HashPassword(secret);
 
await db.ProvisionServiceAccountSubjectAsync(
    subjectId: "service_account::ledger-exporter",
    displayName: "Ledger Exporter",
    clientId: clientId,
    clientSecretHash: secretHash,
    description: "Nightly read-only export for Northwind",
    expiresAt: DateTime.UtcNow.AddDays(90),
    organizationId: "northwind",
    cancellationToken: ct);
 
await db.GrantRoleAsync(
    "service_account::ledger-exporter",
    "ledger::northwind",
    "export_reader",
    ct);
 
await secretManager.StoreAsync(
    "LedgerExporter:Credential",
    $"{clientId}.{secret}",
    ct);
 
await db.SaveChangesAsync(ct);

Never print the credential to stdout, especially from CI or a deployment job where output is retained. Never commit it, write it to normal application logs, or return it from a later read endpoint. If your secret manager has no SDK, use an explicitly interactive one-time admin command with shell history and session recording disabled.

Lower-level alternative: authenticate at your own API boundary#

If OAuth is not appropriate for an internal boundary, the following pattern remains available. It is application-owned and should not be combined with the OAuth credential for the same worker.

3. Authenticate at the API boundary#

Use a dedicated authentication handler or middleware in production. The essential behavior is:

• require TLS;
• parse a bounded credential format;
• look up by ClientId;
• reject missing or expired identities with one generic response;
• verify the stored slow hash;
• attach only the resolved subject identifier to the request.

The protected endpoint below keeps the boundary visible for the example:

using Microsoft.EntityFrameworkCore;
using SqlOS.AuditLogs;
using SqlOS.AuthServer.Services;
using SqlOS.Fga.Interfaces;
using SqlOS.Fga.Models;
 
app.MapPost("/internal/ledger/{organizationId}/export", async (
    string organizationId,
    HttpContext http,
    AppDbContext db,
    ISqlOSFgaAuthService fga,
    ISqlOSAuditLogService auditLogs,
    SqlOSCryptoService crypto,
    CancellationToken ct) =>
{
    if (!TryReadServiceCredential(http, out var clientId, out var secret))
        return Results.Unauthorized();
 
    var account = await db.Set<SqlOSFgaServiceAccount>()
        .SingleOrDefaultAsync(x => x.ClientId == clientId, ct);
 
    if (account is null || account.ExpiresAt is { } expiry && expiry <= DateTime.UtcNow)
        return Results.Unauthorized();
 
    if (!crypto.VerifyPassword(account.ClientSecretHash, secret))
        return Results.Unauthorized();
 
    if (!TryReadIdempotencyKey(http, out var jobKey))
    {
        return Results.BadRequest(new
        {
            message = "A valid Idempotency-Key header is required."
        });
    }
 
    var resourceId = $"ledger::{organizationId}";
    var decision = await fga.CheckAccessAsync(
        account.SubjectId,
        "LEDGER_EXPORT",
        resourceId);
 
    if (!decision.Allowed)
    {
        await RecordExportAuditAsync(
            auditLogs,
            http,
            account,
            organizationId,
            resourceId,
            result: "denied",
            idempotencyKey: $"ledger-export:{jobKey}:denied:{http.TraceIdentifier}",
            ct);
        return Results.Forbid();
    }
 
    account.LastUsedAt = DateTime.UtcNow;
    await db.SaveChangesAsync(ct);
 
    var queued = await exports.EnqueueOnceAsync(
        organizationId,
        account.SubjectId,
        jobKey,
        ct);
 
    await RecordExportAuditAsync(
        auditLogs,
        http,
        account,
        organizationId,
        resourceId,
        result: queued ? "queued" : "already_queued",
        idempotencyKey: $"ledger-export:{jobKey}:outcome",
        ct);
 
    return Results.Accepted(value: new { jobKey, queued });
});

FGA authorization also checks ExpiresAt, so an expired service account cannot retain access through an old grant even if an API boundary accidentally omits its own expiry check. Keep the boundary check anyway so authentication fails before authorization and returns one generic credential error. SqlOSFgaServiceAccount has no separate disabled flag; set ExpiresAt to the current time for immediate authorization revocation, then rotate or remove its credential according to your application policy.

TryReadServiceCredential can accept an app-owned scheme such as Authorization: ServiceAccount <clientId>.<secret>:

static bool TryReadServiceCredential(
    HttpContext http,
    out string clientId,
    out string secret)
{
    clientId = string.Empty;
    secret = string.Empty;
 
    const string prefix = "ServiceAccount ";
    var header = http.Request.Headers.Authorization.ToString();
    if (!header.StartsWith(prefix, StringComparison.Ordinal))
        return false;
 
    var credential = header[prefix.Length..].Trim();
    var separator = credential.IndexOf('.');
    if (separator is < 1 or > 100)
        return false;
 
    clientId = credential[..separator];
    secret = credential[(separator + 1)..];
    return secret.Length is >= 43 and <= 256;
}
 
static bool TryReadIdempotencyKey(HttpContext http, out string jobKey)
{
    jobKey = http.Request.Headers["Idempotency-Key"].ToString().Trim();
    return jobKey.Length is >= 10 and <= 128
        && jobKey.All(character =>
            char.IsAsciiLetterOrDigit(character)
            || character is '-' or '_' or ':' or '.');
}
 
static async Task RecordExportAuditAsync(
    ISqlOSAuditLogService auditLogs,
    HttpContext http,
    SqlOSFgaServiceAccount account,
    string organizationId,
    string resourceId,
    string result,
    string idempotencyKey,
    CancellationToken ct)
{
    await auditLogs.RecordAsync(new SqlOSAuditLogRecordRequest(
        Action: result == "denied"
            ? "ledger.export.denied"
            : "ledger.export.requested",
        OrganizationId: organizationId,
        ApplicationKey: "ledgerops-worker-api",
        Source: "application",
        Actor: new SqlOSAuditActor(
            "service_account",
            account.SubjectId,
            "Ledger Exporter"),
        Targets:
        [
            new SqlOSAuditTarget("ledger", resourceId, organizationId)
        ],
        Context: SqlOSAuditContext.FromHttpContext(http),
        Metadata: new Dictionary<string, object?>
        {
            ["result"] = result
        },
        IdempotencyKey: idempotencyKey),
        ct);
}

Enforce maximum lengths before database work, use fixed-time verification through the password hasher, rate-limit failures, and never echo which half was wrong. Register a log-redaction rule for the whole Authorization header before enabling request logging. EnqueueOnceAsync must enforce a unique job key in durable storage; an in-memory “already seen” check is not sufficient across replicas or restarts.

4. Call the job endpoint#

The scheduler reads its credential at runtime and sends it only to the narrow internal API:

using var request = new HttpRequestMessage(
    HttpMethod.Post,
    "https://api.ledgerops.test/internal/ledger/northwind/export");
 
var runDate = DateOnly.FromDateTime(DateTime.UtcNow);
request.Headers.TryAddWithoutValidation(
    "Idempotency-Key",
    $"nightly-ledger:northwind:{runDate:yyyy-MM-dd}");
 
request.Headers.TryAddWithoutValidation(
    "Authorization",
    $"ServiceAccount {configuration["LedgerExporter:Credential"]}");
 
using var response = await httpClient.SendAsync(request, ct);
response.EnsureSuccessStatusCode();

Do not send the credential through queue payloads, job arguments visible in dashboards, query strings, or telemetry attributes. Reuse the same idempotency key for every retry of one logical run.

When the queue is external, the strongest design stores the durable job intent, audit record, and unique job key in one database transaction/outbox, then publishes asynchronously. The example's EnqueueOnceAsync boundary demonstrates the minimum retry guarantee: an audit or network failure after enqueue cannot create a second export on retry.

5. Rotate without downtime#

For zero-downtime rotation, model two active credentials or add a credential-version table owned by the app:

1. create and store the next secret hash;
2. deploy the new secret to the worker;
3. confirm successful use of the new version;
4. revoke the previous version;
5. record actor, reason, and timestamps in audit logs.

Overwriting ClientSecretHash directly is a single-slot rotation and invalidates the old secret immediately. That is safe only when the worker and API can be changed atomically.

Production checklist#

• Authenticate only over HTTPS and restrict the endpoint at the network layer when possible.
• Use a secret manager and a random 256-bit-or-stronger secret; never use a memorable password.
• Store a salted password hash, not the raw credential.
• Enforce unique credential key IDs in app provisioning; the SqlOS service-account schema does not make ClientId unique.
• Rate-limit failed authentication and alert on sustained failures.
• Give each worker its own subject and credential; never share one identity across jobs.
• Grant the smallest role at the narrowest resource, with EffectiveTo when access is temporary.
• Treat expiry, rotation, revocation, and emergency disable as first-class runbooks.
• Audit successful and denied mutations with the service-account subject ID, never the secret.
• Test that a valid credential with the wrong FGA grant receives 403.
• Test that expired, rotated, malformed, and unknown credentials all receive the same 401 shape.

Expected outcome#

LedgerOps can run unattended without a human token. Compromise of its credential exposes only the one explicitly granted export path, and rotating or expiring that credential does not affect any user session.

Next steps#

• Subject types
• Grants
• Access checks
• Audit logs