Bulk imports of 100+ transactions per chunk hit the default 60s
fastcgi timeout on the main Firefly III vhost too — not just the
importer endpoint. The importer's per-transaction API call to Firefly's
/api/v1/transactions can take 20+s on a fresh DB without ANALYZE,
which compounds with the 30s PHP max_execution_time cap.
- nginx fastcgi_read_timeout=600s on both `firefly` and `firefly-import`
vhosts
- php_admin_value[max_execution_time]=600 + memory_limit=512M on both
PHP-FPM pools
- VANITY_URL on the importer now points to the main Firefly III URL
(was wrongly pointing at the importer's own domain, breaking
clickable transaction-show links in importer log messages)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SQLite was slow under btrfs CoW, and the no-CoW migration path turned
out to be fragile (WAL deletion without checkpoint = data loss). Move
to PostgreSQL on Unix-socket peer auth — no password needed for the
local-host setup, NixOS provisions the database+user declaratively.
Drop the now-unused +C tmpfiles rule on the sqlite directory; the
leftover database.sqlite* files at /var/lib/firefly-iii/storage/database/
are harmless and can be removed manually after switch is verified.
Migration of existing Firefly III data is not preserved by this
commit — fresh-start path: re-register admin, re-issue PAT, re-POST
the bulk CSV through the importer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Random-write SQLite traffic fragments CoW filesystems quickly. The `h`
tmpfiles directive sets +C on the database directory; new SQLite files
(WAL, SHM, recreated main DB) inherit no-CoW automatically. No-op on
non-btrfs filesystems.
Migration of existing files must be done manually with checkpoint-first:
systemctl stop phpfpm-firefly-iii.service
sqlite3 .../database.sqlite 'PRAGMA wal_checkpoint(TRUNCATE);'
# then recreate main file inside the +C dir
systemctl start phpfpm-firefly-iii.service
Skipping the wal_checkpoint and naively deleting .sqlite-wal will lose
all writes that haven't been checkpointed (PHP-FPM SIGTERM does not
trigger a checkpoint).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end verified: aqbanking-cli fetches Sparda Südwest transactions
via FinTS PIN/TAN + SecureGo+, exports CSV using a custom decimal-amount
profile, POSTs to firefly-iii-data-importer's autoupload endpoint, which
creates transactions in Firefly III via API.
Changes vs. previous WIP commit:
- firefly/access_token sops slot for the importer's Firefly III API auth
(FIREFLY_III_ACCESS_TOKEN_FILE — was the missing piece causing 401s
from the API after the autoupload secret authenticated)
- nginx fastcgi_read_timeout=600s on the importer vhost (prevents 504
while PHP-FPM is still processing the batch)
- PHP-FPM max_execution_time=600s + memory_limit=512M on the importer
pool (PHP's stock 30s aborts mid-import for batches > ~50 transactions)
- timer re-enabled, wantedBy=[timers.target]
Caveats baked into a code comment:
- Sparda online-banking PIN must be [A-Za-z0-9] only. aqbanking 6.8.2's
-P pinfile mangles `:`, `+`, `'`, `?`, `@`, `%`, `*`; bank locks the
access (3 soft / 9 hard strikes) on rejected attempts. Same applies
whenever the sops secret is rotated.
- Bulk historical imports beyond the PSD2 90-day window need interactive
SCA approval per ~30-day chunk and cannot run from the timer; the
daily 35-day rolling window stays inside the no-SCA region.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
End-to-end FinTS pipeline against Sparda Südwest is wired up but
disabled — aqbanking 6.8.2's `-P pinfile` flag does not consume the
file content correctly on this build (verified: pinfile bytes match
the manually-typed PIN exactly, yet the bank receives a wrong PIN).
Three rejected attempts locked the access at Sparda; do not re-arm
the timer until the auth path is replaced (likely python-fints).
What works:
- aqbanking config and FinTS dialog (manual PIN entry)
- getaccsepa workaround for HKCAZ "Mussfeld 9160" rejection
- custom CSV profile (decimal amounts + IBAN columns) wired in
- Firefly importer auto-upload settings + sops secret slot
- inbox + profile-symlink tmpfiles
What's broken:
- Headless PIN delivery via aqbanking-cli -P
- Timer left wantedBy=[] so it cannot fire post-deploy
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lays the groundwork for Sparda-Bank Südwest transaction sync via
direct FinTS (no third-party data proxy). aqbanking-cli in the system
PATH, persistent state at /var/lib/firefly-aqbanking, sops slot for
the online-banking PIN. Initial enrollment must be done interactively
on the host; systemd timer for automated fetches comes in a follow-up.
