feat(sgx): finish firefly-sparda-fetch — headless FinTS import

End-to-end verified: aqbanking-cli fetches Sparda Südwest transactions
via FinTS PIN/TAN + SecureGo+, exports CSV using a custom decimal-amount
profile, POSTs to firefly-iii-data-importer's autoupload endpoint, which
creates transactions in Firefly III via API.

Changes vs. previous WIP commit:
- firefly/access_token sops slot for the importer's Firefly III API auth
  (FIREFLY_III_ACCESS_TOKEN_FILE — was the missing piece causing 401s
  from the API after the autoupload secret authenticated)
- nginx fastcgi_read_timeout=600s on the importer vhost (prevents 504
  while PHP-FPM is still processing the batch)
- PHP-FPM max_execution_time=600s + memory_limit=512M on the importer
  pool (PHP's stock 30s aborts mid-import for batches > ~50 transactions)
- timer re-enabled, wantedBy=[timers.target]

Caveats baked into a code comment:
- Sparda online-banking PIN must be [A-Za-z0-9] only. aqbanking 6.8.2's
  -P pinfile mangles `:`, `+`, `'`, `?`, `@`, `%`, `*`; bank locks the
  access (3 soft / 9 hard strikes) on rejected attempts. Same applies
  whenever the sops secret is rotated.
- Bulk historical imports beyond the PSD2 90-day window need interactive
  SCA approval per ~30-day chunk and cannot run from the timer; the
  daily 35-day rolling window stays inside the no-SCA region.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Harald Hoyer 2026-05-01 18:58:45 +02:00
parent 74af9fd5ac
commit e0d2a2f50d

View file

@ -40,6 +40,10 @@ in
sopsFile = ../../../.secrets/sgx/firefly.yaml;
owner = "firefly-iii-data-importer";
};
"firefly/access_token" = {
sopsFile = ../../../.secrets/sgx/firefly.yaml;
owner = "firefly-iii-data-importer";
};
};
environment.systemPackages = [ pkgs.aqbanking ];
@ -114,12 +118,13 @@ in
'';
};
# Timer disabled while we work around aqbanking 6.8.2's broken
# `-P pinfile` handling. The fetch service authenticates with a wrong
# PIN against the bank — three runs locked the access at Sparda. Do
# not re-enable until the auth path is replaced (likely python-fints).
# Sparda online-banking PIN must contain only [A-Za-z0-9] — special
# chars (`:`, `+`, `'`, `?`, `@`, `%`, `*`) get mangled by aqbanking
# 6.8.2's pinfile path and the bank locks the access after a few
# rejected attempts (3 soft / 9 hard). Same applies if the secret in
# sops is rotated.
timers.firefly-sparda-fetch = {
wantedBy = [ ];
wantedBy = [ "timers.target" ];
timerConfig = {
OnCalendar = "daily";
Persistent = true;
@ -157,12 +162,27 @@ in
CAN_POST_AUTOIMPORT = "true";
IMPORT_DIR_ALLOWLIST = inbox;
AUTO_IMPORT_SECRET_FILE = config.sops.secrets."firefly/auto_import_secret".path;
FIREFLY_III_ACCESS_TOKEN_FILE = config.sops.secrets."firefly/access_token".path;
};
};
nginx.virtualHosts = {
${domain} = vhostBase;
${importDomain} = vhostBase;
# Importer's autoupload endpoint blocks until the entire batch
# finishes — POSTing 100+ transactions takes minutes. Default 60s
# fastcgi timeout makes nginx 504 even though PHP-FPM keeps going.
${importDomain} = vhostBase // {
extraConfig = ''
fastcgi_read_timeout 600s;
'';
};
};
# PHP's stock max_execution_time = 30s aborts large bulk imports
# mid-stream. Match the nginx fastcgi_read_timeout above.
phpfpm.pools.firefly-iii-data-importer.settings = {
"php_admin_value[max_execution_time]" = "600";
"php_admin_value[memory_limit]" = "512M";
};
};