# Gotchas, with fixes, from The Stack > 413 things that bit Nicolas Dao (Cloudless Labs, Sydney) or his clients in production, across 97 cloud, edge, AI and dev tools, each with its fix. Each heading links the tool's full entry. - Web page: https://cloudlesslabs.com/stack/gotchas/ ## Ahrefs, Semrush and other SEO subscription suites [Ahrefs, Semrush and other SEO subscription suites](https://cloudlesslabs.com/stack/seo-suites/index.md): Skip. - **Gotcha**: The price is per user: every extra seat is another $40 to $80 a month on Ahrefs, or from $45 on Semrush. **Fix**: Keep one seat for the person who lives in it, and give everyone else the reports. Or run the research through DataForSEO and share the agent's write-up. - **Gotcha**: Ahrefs' Lite plan caps usage at 1,000 credits per user, and limits history to 6 months. **Fix**: Check the plan's limits against the work before you buy. For a one-off audit, a few dollars of DataForSEO calls is the cheaper path. ## Amazon Bedrock [Amazon Bedrock](https://cloudlesslabs.com/stack/aws-bedrock/index.md): Watching. - **Gotcha**: The default and the global endpoint route anywhere in the world. Australian residency only happens if you ask for it. **Fix**: Call from ap-southeast-2 or ap-southeast-4 with the au. inference profile (or Melbourne's in-region endpoint), and block the other Regions with a service control policy. - **Gotcha**: AWS warns that a profile's geographic prefix alone doesn't prove where requests go, and destinations can differ by source Region. **Fix**: Run get-inference-profile from every source Region you use, and keep the list of destination Regions as evidence for the client's compliance file. - **Gotcha**: Not every Claude model has an Australian route, and new models often reach the US first. **Fix**: Check the model's Regional availability before you promise residency, and pin the model ID: don't let "latest" move you to a model that routes overseas. - **Gotcha**: Prompts and outputs can be stored in opt-in Regions for abuse detection when a profile routes there. **Fix**: Use the AU profile, whose destinations are only Sydney and Melbourne, and check Bedrock's data-protection terms against the client's obligations. ## Amazon CloudFront [Amazon CloudFront](https://cloudlesslabs.com/stack/aws-cloudfront/index.md): Use when it fits. - **Gotcha**: Pay-as-you-go egress can turn a viral page into a big bill. **Fix**: Check the flat-rate plans, or put Cloudflare in front. - **Gotcha**: Flat-rate plans never charge overage, but they aren't unlimited. After a one-off spike of up to 3x the allowance, sustained use above it can get your traffic served from fewer, more distant edge locations. **Fix**: Watch the monthly allowance notifications and move up a tier when the baseline grows. Put large public files on R2 rather than inside the plan. - **Gotcha**: The certificate for a custom domain must be in ACM in us-east-1, even when the whole app is in Sydney. A certificate created in ap-southeast-2 can't be attached. **Fix**: In Pulumi, create the certificate with a second AWS provider pinned to us-east-1 and pass its ARN to the distribution. - **Gotcha**: With origin access control and no s3:ListBucket permission, S3 answers 403 for a missing file. SPA deep links and broken links show Access Denied instead of a page. **Fix**: For an SPA, map 403 and 404 to /index.html with a 200 in the distribution's custom error responses. For a normal site, grant CloudFront s3:ListBucket so missing files return a real 404. - **Gotcha**: Invalidations are free for the first 1,000 paths a month across the account, then $0.005 per path. A deploy script that invalidates every changed file one by one runs past that quickly. **Fix**: Use content-hashed file names for assets and invalidate only the HTML, or send one /* path: a wildcard counts as one path. ## Amazon CloudWatch [Amazon CloudWatch](https://cloudlesslabs.com/stack/aws-cloudwatch/index.md): My default. - **Gotcha**: Cost Explorer labels log ingestion 'PutLogEvents', which looks like a per-call charge. It's billed per GB ingested. **Fix**: Find the log groups with the most IncomingBytes. The cost is in bytes, not calls. - **Gotcha**: Logging whole message payloads in high-throughput functions. One function produced 59% of a client's 1 TB a month. **Fix**: Log IDs and errors, not payloads. If payloads must be kept, send them to S3 or sample them. - **Gotcha**: The default Text log format can't drop DEBUG and INFO lines before ingestion. **Fix**: Switch Lambda to JSON log format with application log level WARN in production. - **Gotcha**: Log groups never expire by default. **Fix**: Set retention on every log group in your Pulumi code. - **Gotcha**: Infrequent Access log groups are half price but drop Live Tail, metric filters, subscription filters and Embedded Metric Format, which silently breaks Powertools Metrics. A log group's class can't be changed later. **Fix**: Use IA only for verbose groups you rarely read, and create a new group (repoint LoggingConfig) to switch. - **Gotcha**: An alerting pipeline that stops working looks exactly like a quiet night: no alarms either way. **Fix**: Alarm on the alarm queue's ApproximateAgeOfOldestMessage and on the alarm function's errors and throttles, with missing data treated as breaching. Send a scheduled drill alarm through the whole path, and alarm if its acknowledgement doesn't come back. - **Gotcha**: API Gateway execution logs at INFO level log every request. **Fix**: Keep access logs, turn execution logs down. ## Amazon Cognito [Amazon Cognito](https://cloudlesslabs.com/stack/aws-cognito/index.md): My default. - **Gotcha**: Cognito gives trigger Lambdas about 5 seconds. Cold container-image Lambdas blow through it and sign-in fails. **Fix**: Keep triggers small and zip-packaged. - **Gotcha**: Cognito's built-in email sender is capped at 50 emails a day per AWS account. **Fix**: Point the user pool at SES before going live. - **Gotcha**: Passkeys need the username first, and can only be registered after a first sign-in. **Fix**: Offer passkeys as an upgrade after sign-in, not as the first-time path. ## Amazon Data Firehose [Amazon Data Firehose](https://cloudlesslabs.com/stack/aws-firehose/index.md): My default. - **Gotcha**: Direct PUT bills every record rounded up to 5 KB. One 200-byte log line per record costs 25 times the list price, about $925 per TB. **Fix**: Pack many newline-delimited lines into each record (up to about 1,000 KB). Records of 100 KB or more make the rounding negligible. - **Gotcha**: Lambda freezes the environment when the handler returns, so buffered lines can be lost. **Fix**: Await a flush before returning (try/finally), flush on size, and send WARN/ERROR synchronously to console so CloudWatch keeps the critical lines. - **Gotcha**: A stream accepts 1 MiB/s by default in Sydney. Busy peaks throttle. **Fix**: Request a higher limit up front, and retry the records reported in FailedPutCount. - **Gotcha**: Lambda's built-in log destination to Firehose is not the cheap path: CloudWatch charges $0.335/GB to deliver it, then Firehose charges $0.17/GB for vended logs. **Fix**: Call PutRecordBatch from your own logger instead. - **Gotcha**: Dynamic partitioning looks handy and adds about $25 per TB plus per-object and per-hour fees. **Fix**: Use the default YYYY/MM/DD/HH prefix (free) and one stream per app or per table. ## Amazon DynamoDB [Amazon DynamoDB](https://cloudlesslabs.com/stack/aws-dynamodb/index.md): Never. - **Gotcha**: Reads are eventually consistent by default, so a read right after a write can miss it. **Fix**: Use strongly consistent reads (tables only, not global secondary indexes), or use Postgres. - **Gotcha**: A Query or Scan returns at most 1 MB per call. Code that reads one page and stops silently misses the rest. **Fix**: Always loop on LastEvaluatedKey until it's empty, and design keys so a Query reads one small item collection. - **Gotcha**: One partition handles at most 1,000 write units and 3,000 read units a second. A hot key (one tenant, one counter, today's date) gets throttled even when the table is far below its limits. **Fix**: Spread writes over several key values (write sharding) and aggregate on read, or keep hot counters somewhere built for them. - **Gotcha**: Items are capped at 400 KB, including attribute names, and a transaction can touch at most 100 items and 4 MB. **Fix**: Store large documents in S3 and keep the key in DynamoDB. Keep transactions small, or use a database with real transactions. ## Amazon EventBridge [Amazon EventBridge](https://cloudlesslabs.com/stack/aws-eventbridge/index.md): My default. - **Gotcha**: Scheduled rules on the event bus (now labelled legacy) only run in UTC, so a 9 am Sydney job drifts by an hour twice a year. **Fix**: Use EventBridge Scheduler and set the schedule's time zone (scheduleExpressionTimezone in Pulumi) to Australia/Sydney, or the client's zone. - **Gotcha**: Scheduler follows daylight saving: a cron time inside the spring-forward gap is skipped that day, and the repeated hour in autumn runs only once. In Sydney that gap is 2:00–2:59 am on the first Sunday of October. **Fix**: Don't schedule daily jobs between 2 and 3 am local time, or schedule them in UTC if they must run every single day. - **Gotcha**: Delivery is at least once. A target can receive the same event or schedule twice, and the classic bus doesn't keep order. **Fix**: Make every target idempotent: use the event ID or a business key, and ignore what you've already processed. - **Gotcha**: A failed delivery is retried for up to 24 hours (185 attempts by default), then dropped silently. **Fix**: Attach an SQS dead-letter queue to every rule target and schedule, and alarm on its depth. - **Gotcha**: One-time schedules keep counting against your quota after they've run. **Fix**: Set ActionAfterCompletion to DELETE on one-time schedules. ## Amazon Kinesis [Amazon Kinesis](https://cloudlesslabs.com/stack/aws-kinesis/index.md): Use when it fits. - **Gotcha**: One bad record blocks its shard. With default settings, Lambda retries the failing batch until the records expire: up to a week stuck. **Fix**: On the event source mapping, set MaximumRetryAttempts, MaximumRecordAgeInSeconds and BisectBatchOnFunctionError, return ReportBatchItemFailures, and send failures to an S3 on-failure destination (it keeps the payload). - **Gotcha**: PutRecords returns HTTP 200 even when some records failed. They're only listed in FailedRecordCount. **Fix**: Check FailedRecordCount on every call and retry just the failed entries with backoff. - **Gotcha**: On-demand streams start at 4 MB/s of writes and throttle when traffic more than doubles its previous peak within 15 minutes. **Fix**: Retry throttled records. Before a known spike, switch to provisioned with enough shards, or set warm throughput (On-demand Advantage only). - **Gotcha**: A single partition key is capped at one shard: 1 MB/s or 1,000 records a second, in any mode. **Fix**: Partition on something with many values (device ID, account ID), never on a type or a date. - **Gotcha**: Standard consumers share each shard's 2 MB/s and five reads a second. A third consumer starts starving the others. **Fix**: Beyond one or two consumers, register enhanced fan-out consumers: each gets its own throughput, for an extra per-GB charge. ## Amazon OpenSearch [Amazon OpenSearch](https://cloudlesslabs.com/stack/aws-opensearch/index.md): Skip. - **Gotcha**: The index is eventually consistent: a document becomes searchable only after the next refresh (every second by default, 30 seconds or more if you follow AWS's tuning advice). A user saves a record and doesn't find it. **Fix**: Use refresh=wait_for on the writes a user will search for straight away, and never treat OpenSearch as the source of truth. - **Gotcha**: It's a second copy of your data. Dual writes from the app drift the first time one of them fails. **Fix**: Feed it one way from the database (an outbox table or change stream into a queue), and keep a script that rebuilds every index from the source. - **Gotcha**: t2 and t3.small nodes become unstable under sustained load, and a domain without dedicated master nodes can go unresponsive for a few seconds after an upgrade while it elects a master. **Fix**: For production: three Availability Zones, three dedicated master nodes, and r6g.large or newer data nodes. Budget for it before you choose OpenSearch. - **Gotcha**: Shard size drives performance. Too many small shards waste memory; huge ones slow recovery. **Fix**: Aim for 10–30 GiB per shard for search workloads, a shard count that's a multiple of the data nodes, and index templates that set it. - **Gotcha**: A classic Serverless collection never drops below its minimum OCUs, so an idle dev environment still costs hundreds of dollars a month. **Fix**: Use a NextGen collection (it scales to zero when idle), dev-test mode, or a single small managed domain for non-production. ## Amazon Pinpoint [Amazon Pinpoint](https://cloudlesslabs.com/stack/aws-pinpoint/index.md): Never. - **Gotcha**: Existing Pinpoint campaigns and journeys stop being supported on 30 October 2026. **Fix**: Move marketing email to Resend (or SES with your own list tool). - **Gotcha**: After 30 October 2026 the console, endpoints, segments and analytics APIs are gone, and the KPI history with them. **Fix**: Export now: endpoints through a segment with no filters exported to S3, campaigns and journeys with the get-campaign and get-journey APIs, and the last three months of KPIs with the date-range KPI APIs. - **Gotcha**: Sending to a stored endpoint ID, and SendUsersMessages, stop working for every channel, push included. **Fix**: Change the code to send to explicit addresses and device tokens with SendMessages, or move push to SNS mobile push. - **Gotcha**: Pinpoint hides in other services: a Cognito user pool app client can be wired to a Pinpoint project for sign-in analytics. **Fix**: Search every account and stack for Pinpoint project ARNs (Cognito AnalyticsConfiguration, Amplify analytics) and remove them before the cut-off. ## Amazon Quick Sight [Amazon Quick Sight](https://cloudlesslabs.com/stack/aws-quick-sight/index.md): Never. - **Gotcha**: Dashboards built by clicking in the console live outside the repository: no review, no history next to the code, and no clean way to rebuild them in another environment. **Fix**: If you must keep it, export and version the definitions through its APIs. Better: build the dashboards in your app, as code. ## Amazon S3 [Amazon S3](https://cloudlesslabs.com/stack/aws-s3/index.md): My default. - **Gotcha**: An upload that never completes leaves its parts in the bucket, billed as storage, and invisible in a normal object listing. **Fix**: Add a lifecycle rule with AbortIncompleteMultipartUpload (7 days is AWS's own example) to every bucket. - **Gotcha**: With versioning on, deleting or overwriting an object keeps the old version, and you keep paying for it. **Fix**: Add a lifecycle rule that expires noncurrent versions after the retention you actually need (for example 30 days). - **Gotcha**: S3 has no locking between concurrent writers: two PUTs to the same key, and the last one wins. **Fix**: Use conditional writes: If-None-Match to create only if the key doesn't exist, If-Match with the ETag to update only what you read. A loser gets 412 and retries. - **Gotcha**: Each prefix handles at least 3,500 writes and 5,500 reads a second, and S3 scales up gradually. A sudden burst on one prefix gets 503 Slow Down errors. **Fix**: Spread hot keys across prefixes (for example by date and ID), and let the SDK retry with backoff. - **Gotcha**: Standard-IA and Glacier Instant Retrieval bill every object as at least 128 KB, for at least 30 or 90 days. A lifecycle rule that moves millions of small files there can raise the bill. **Fix**: Transition only large, old objects (filter by size in the rule), or use Intelligent-Tiering, which doesn't charge the minimums and leaves objects under 128 KB in its frequent tier. ## Amazon S3 Vectors [Amazon S3 Vectors](https://cloudlesslabs.com/stack/aws-s3-vectors/index.md): Use when it fits. - **Gotcha**: Every query is billed as if it read the whole index (each vector, its key and its filterable metadata), so the bill grows with queries times index size, not with storage. **Fix**: Declare metadata you never filter on as non-filterable (it isn't counted), split huge collections into one index per client or per period, and keep hot, frequent search in pgvector. - **Gotcha**: No hybrid (keyword + vector) search. **Fix**: Search pgvector and S3 Vectors separately and merge with Reciprocal Rank Fusion, or put OpenSearch's s3vector engine on top at scale. ## Amazon SES [Amazon SES](https://cloudlesslabs.com/stack/aws-ses/index.md): Use when it fits. - **Gotcha**: New SES accounts silently default to the Essentials plan at $0.16 per 1,000, 60% more than à la carte. **Fix**: Choose the à la carte plan explicitly if you don't need Essentials features. - **Gotcha**: Sandbox exit is per region, and AWS quotes about 96 hours for a first response. **Fix**: Request production access on day one of the project, in every region you'll send from. - **Gotcha**: The Essentials default also applies to an existing account the first time it uses SES in a new region (no SES activity there since 1 June 2025). **Fix**: Run aws sesv2 put-account-pricing-attributes --plan NONE in each new region. The first switch away from a default plan takes effect immediately. - **Gotcha**: Reputation is per account and region. At 5% bounces or 0.1% complaints the account goes under review; at 10% or 0.5% AWS can pause sending, for every app in it. **Fix**: Keep marketing in a separate account from transactional mail, keep the account-level suppression list on, and alarm on Reputation.BounceRate and Reputation.ComplaintRate. - **Gotcha**: Gmail doesn't send complaint data to SES, so Gmail users who hit Spam never reach your suppression list. **Fix**: Add a one-click List-Unsubscribe header to anything bulk, and drop addresses that never open or click. ## Amazon SNS [Amazon SNS](https://cloudlesslabs.com/stack/aws-sns/index.md): Use when it fits. - **Gotcha**: SNS → Lambda invokes a function per message: at IoT volumes that's millions of tiny invocations. **Fix**: SNS → SQS → Lambda with batching, or skip SNS and write straight to SQS. - **Gotcha**: In Sydney, an account can publish only 1,500 messages a second by default, across all its standard topics (US East gets 30,000). Above that, publishes are throttled. **Fix**: Raise the quota in Service Quotas before launch. For sustained high-volume ingestion, write to SQS or Kinesis instead. - **Gotcha**: When a subscriber stays unreachable, SNS retries, then discards the message. With Lambda it's worse: after the hand-off, Lambda retries a failing function twice, then drops the event. **Fix**: Give every subscription a dead-letter queue (RedrivePolicy), and every subscribed Lambda an on-failure destination. Better: subscribe SQS queues, not functions. - **Gotcha**: Payload-based filter policies are billed per GB scanned, for every subscription that has one, matched or not. **Fix**: Put the routing fields in message attributes and filter on those: attribute filtering is free. - **Gotcha**: Standard topics order messages on a best-effort basis and deliver at least once: expect the odd duplicate or out-of-order message. **Fix**: Make consumers idempotent (a message ID as a unique key). If order matters, use a FIFO topic into FIFO queues. ## Amazon SQS [Amazon SQS](https://cloudlesslabs.com/stack/aws-sqs/index.md): My default. - **Gotcha**: Messages reappear and get processed twice, or go to the DLQ, when the queue's visibility timeout is too close to the function's timeout. **Fix**: Set the visibility timeout to at least six times the function timeout. - **Gotcha**: One bad message in a batch fails the whole batch, and the good messages retry too. **Fix**: Enable ReportBatchItemFailures and return only the failed message IDs. - **Gotcha**: Messages bounce to the DLQ after too few attempts, or loop forever. **Fix**: Set maxReceiveCount deliberately, at least 5 as AWS recommends, and alarm on DLQ depth. - **Gotcha**: An urgent message waits seconds in a queue tuned for throughput: the batching window holds it, and a capped consumer works through the backlog first. **Fix**: Give urgent messages their own queue, read with batch size 1, no batching window, and its own function. Alarm on ApproximateAgeOfOldestMessage, not only on depth. ## Amazon Web Services (AWS) [Amazon Web Services (AWS)](https://cloudlesslabs.com/stack/aws/index.md): My default. - **Gotcha**: IAM policies written by hand end up either too broad ("Action": "*") or so tight that every deploy fails on a missing permission. **Fix**: Have the agent write the policy from the code, validate it with IAM Access Analyzer's policy checks, and give people access through IAM Identity Center permission sets rather than IAM users. - **Gotcha**: Security groups are stateful and only allow; network ACLs are stateless, allow and deny, and need the return traffic's ephemeral ports opened. Tightening a NACL silently breaks responses. **Fix**: Leave NACLs at their default and do the work in security groups, with rules that reference other security groups instead of IP ranges. - **Gotcha**: There's no built-in rate dial between a queue and its consumer. A spike on SQS spins up Lambdas until the database downstream falls over. **Fix**: Set maximum concurrency on the SQS event source mapping, set the visibility timeout to at least six times the function timeout, and send failures to a DLQ with a deliberate maxReceiveCount. - **Gotcha**: Budgets only send alerts. Nothing stops spending on its own. **Fix**: Attach budget actions that apply a deny policy or stop instances, set thresholds below the real limit, and review Cost Explorer monthly. ## Anthropic API (for client systems) [Anthropic API (for client systems)](https://cloudlesslabs.com/stack/anthropic-api/index.md): Skip. - **Gotcha**: Claude 4.7 and later models use a newer tokenizer that, by Anthropic's own docs, produces about 30% more tokens for the same text than the previous one. A per-token price comparison on paper understates the real bill. **Fix**: Compare on real traffic: send a week of the client's actual prompts to both models through OpenRouter and compare the billed cost of the answers that passed your checks. - **Gotcha**: There's no Australian option on the first-party API. inference_geo accepts only us (at 1.1x the price) or global, and workspace data is stored in the US. **Fix**: For Australian residency, call Claude through Amazon Bedrock in ap-southeast-2 with the AU inference profile, and budget the 10% regional premium. - **Gotcha**: Claude in Amazon Bedrock doesn't support everything the first-party API does: no Batch API, no Files API, no server-side web search or code execution, no MCP connector, no Managed Agents. **Fix**: Check the feature list before you pick Bedrock for residency. Run batch jobs yourself (SQS and Lambda), and call your own tools instead of server-side ones. - **Gotcha**: Prompt caching only saves money when the start of the prompt repeats. A cache write costs more than plain input (1.25x for five minutes, 2x for an hour), so a prompt that changes at the top pays the premium and never hits. **Fix**: Put the fixed parts first (tools, system prompt, reference documents) and the variable part last, and use the one-hour cache only when the same prefix is read at least twice within the hour. ## API Gateway [API Gateway](https://cloudlesslabs.com/stack/aws-api-gateway/index.md): Use when it fits. - **Gotcha**: REST API pricing ($3.50/M) is almost three times HTTP API pricing ($1.29/M). **Fix**: Default to HTTP APIs, or a function URL, unless you need a REST-only feature. - **Gotcha**: Adding API Gateway later looks like a rewrite. **Fix**: Not behind an HTTP API: it uses the same event format as a function URL (payload version 2.0), so the same handler works behind either. A REST API sends the older 1.0 event, so moving there needs a thin adapter in the handler. - **Gotcha**: API Gateway execution logs at INFO level log every request into CloudWatch. **Fix**: Keep access logs; turn execution logging down or off in production. ## Astro on Cloudflare Workers [Astro on Cloudflare Workers](https://cloudlesslabs.com/stack/astro-on-workers/index.md): Use when it fits. - **Gotcha**: Astro 6 changed how the Cloudflare adapter exposes bindings: `Astro.locals.runtime` is deprecated. **Fix**: Read bindings (KV, R2, secrets) through `env` instead, and check the adapter's upgrade notes when you move between major versions. ## Aurora PostgreSQL [Aurora PostgreSQL](https://cloudlesslabs.com/stack/aws-aurora/index.md): Use when it fits. - **Gotcha**: RDS Proxy 'pins' a session to one connection when it sees SET, prepared statements, temporary tables, advisory locks or statements over 16 KB. Pinned sessions exhaust the pool. **Fix**: Watch the DatabaseConnectionsCurrentlySessionPinned metric and remove the pinning causes. Or move HTTP logic to Lambda Web Functions, which share one pool across up to 128 requests, or to the Data API, which has no pinning. - **Gotcha**: Serverless v2 can pause at 0 ACU, but resuming takes about 15 seconds (30+ after a day paused). **Fix**: Don't pause production databases that serve users. - **Gotcha**: An RDS Proxy keeps a connection open to every instance, so a Serverless v2 cluster with a proxy never pauses at 0 ACU. **Fix**: Use the Data API on clusters that should pause, such as dev, test and internal tools. - **Gotcha**: The Data API terminates any call that returns more than 1 MiB, and rejects a row over 64 KB. **Fix**: Page with LIMIT and keyset pagination, select only the columns you need, and keep large documents in S3 with a key in the row. For exports, use a normal driver. - **Gotcha**: The Data API only talks to the writer, even for reads. Readers added for read scale get none of its traffic, and a Global Database secondary fails every call until it's promoted. **Fix**: Send read-heavy traffic to the reader endpoint with a normal driver. Keep the Data API for the app's own small reads and writes. - **Gotcha**: The Data API isn't supported on T instance classes (db.t3, db.t4g), the cheap provisioned size for dev. **Fix**: Use Serverless v2 for small clusters that need the Data API. - **Gotcha**: A Data API statement is cancelled after 45 seconds, and an AWS SDK with a shorter timeout (Java v2: 30 seconds) can retry a statement that already ran: a duplicate INSERT. **Fix**: Set the SDK's call and socket timeouts to at least 60 seconds, make writes idempotent with a unique key, and use continueAfterTimeout for long DDL. - **Gotcha**: A Data API transaction is rolled back after 3 minutes without a call, and after 1 hour in any case. Two calls on the same transaction ID at once fail. **Fix**: Keep transactions short and send their statements one after another, never with Promise.all. - **Gotcha**: The Data API reuses connections and runs DISCARD ALL between calls: SET, prepared statements and temporary tables don't survive, but state held by an extension such as dblink can leak into the next call. **Fix**: Put session settings inside a transaction, and open and close extension state within it. - **Gotcha**: A call to a paused Serverless v2 writer fails with DatabaseResumingException while Aurora wakes it. **Fix**: Retry with backoff for about 30 seconds. Don't auto-pause a database that serves users. ## AWS Backup [AWS Backup](https://cloudlesslabs.com/stack/aws-backup/index.md): My default. - **Gotcha**: A Vault Lock in compliance mode becomes permanent once its grace time (at least 3 days) ends. Nobody, not the root user and not AWS, can remove it or delete a recovery point early, and anything kept 'Always' is kept forever. **Fix**: Start in governance mode, check every rule's retention, then switch to compliance. Never set 'Always' retention in a locked vault. - **Gotcha**: Cross-account copies of RDS, Aurora, EBS and other types not fully managed by AWS Backup fail with AWS-managed KMS keys, because those keys can't be shared. The default vault can't be a destination either. **Fix**: Encrypt the source with a customer-managed KMS key shared with the backup account, and copy into a dedicated vault there. - **Gotcha**: RDS continuous backups (point-in-time restore) don't copy to another account or region. AWS Backup copies a snapshot instead. **Fix**: Plan two recovery points: point-in-time restore in the production account for 'undo last hour', daily snapshot copies in the backup account for 'the account is gone'. - **Gotcha**: AWS Backup only touches the resource types you've opted in to, per Region. A plan that selects everything by tag still skips a type that isn't opted in. **Fix**: Set the opt-in in code (aws.backup.RegionSettings) for every Region you use. - **Gotcha**: Closing an account suspends it for 90 days, then AWS deletes its vaults, even locked ones. **Fix**: Keep the backup account out of any account clean-up, and deny organizations:LeaveOrganization on it with an SCP. ## AWS CloudFormation [AWS CloudFormation](https://cloudlesslabs.com/stack/aws-cloudformation/index.md): Never. - **Gotcha**: A failed update can roll back into UPDATE_ROLLBACK_FAILED, usually because something in the stack was changed or deleted outside CloudFormation. The stack can't be updated until it's fixed. **Fix**: Fix the resource by hand, then continue the rollback (continue-update-rollback), skipping resources that can't recover. Never touch stack resources outside CloudFormation. - **Gotcha**: A template holds at most 500 resources, and 51,200 bytes inline (1 MB from S3). Real systems hit it. **Fix**: Split into nested stacks, or stop fighting the format and move the stack to Pulumi. - **Gotcha**: Deleting a stack fails on an S3 bucket that still has objects in it, and leaves the stack in DELETE_FAILED. **Fix**: Empty the bucket first, or delete again with RetainResources and remove the bucket yourself. - **Gotcha**: Some vendors (monitoring and security tools, AWS's own solutions) only ship their AWS integration as a CloudFormation template. **Fix**: Deploy the template from Pulumi with aws.cloudformation.Stack. It stays one opaque resource in your code, and the rest stays in real code. - **Gotcha**: A client already has years of CloudFormation stacks. **Fix**: Migrate gradually: read their outputs from Pulumi, then import the resources (the import option in code) stack by stack, and delete each stack with its resources retained. ## AWS Database Migration Service [AWS Database Migration Service](https://cloudlesslabs.com/stack/aws-dms/index.md): My default. - **Gotcha**: LOBs are cut silently. Limited LOB mode is the default and stops at 32 KB, and DMS treats Postgres JSON as a LOB, so large JSON documents arrive truncated. **Fix**: Measure the largest LOB and JSON values first and set Max LOB size above them, or use inline or full LOB mode for those tables. Search the task log for truncation warnings. - **Gotcha**: DMS creates tables and primary keys, nothing else: no secondary indexes, foreign keys, users or permissions. **Fix**: Create the target schema first (Schema Conversion, or the engine's own tools). Add secondary indexes before change capture starts, and foreign keys and triggers just before cut-over. - **Gotcha**: On a Postgres source, the replication slot keeps every WAL file until DMS reads it. A stopped or idle task lets WAL pile up until the source disk fills. **Fix**: Turn on HeartbeatEnable in the Postgres endpoint settings, alarm on disk and replication-slot lag, and drop the slot when the migration is done. - **Gotcha**: Tables without a primary key: DMS ignores their UPDATE and DELETE operations during change capture, so the target drifts. **Fix**: Add primary keys before migrating. If you can't, set REPLICA IDENTITY FULL on those tables, after testing the extra WAL it generates. - **Gotcha**: Truncated values and rows rejected by foreign keys appear only in the task log, not in the console status. **Fix**: Send task logs to CloudWatch and turn on data validation, which compares source and target after the full load. ## AWS End User Messaging [AWS End User Messaging](https://cloudlesslabs.com/stack/aws-end-user-messaging/index.md): Skip. - **Gotcha**: New accounts start in a sandbox: $1 of SMS a month to verified numbers, and 20 voice calls a day. **Fix**: Open the support case to leave the sandbox at the start of the project. - **Gotcha**: An Australian long code takes about three weeks, and a number that can call needs a separate support case. **Fix**: Request numbers early, and confirm voice capability in writing. - **Gotcha**: Each country's messages go through a single carrier partner: one point of failure. **Fix**: For anything critical, keep a second provider ready. - **Gotcha**: A call that reaches voicemail reports COMPLETED, the same as a person listening. **Fix**: Don't use it where you need to know a person heard the message. - **Gotcha**: Pinpoint reaches end of support on 30 October 2026. Its SMS and voice channels live on as End User Messaging, but the old Pinpoint APIs don't. **Fix**: Move Pinpoint SMS code to the pinpoint-sms-voice-v2 API. ## AWS IoT Core [AWS IoT Core](https://cloudlesslabs.com/stack/aws-iot-core/index.md): Use when it fits (Proposed: research-backed, not yet confirmed by Nic). - **Gotcha**: Messages, rules and actions are all metered in 5 KB steps. An 8 KB reading through a rule with two actions bills as two rules and four actions. **Fix**: Keep readings under 5 KB, and give each rule one action (SQS). Fan out later from SQS or EventBridge if needed. - **Gotcha**: If a rule action fails (missing permission, throttled queue), the message is dropped unless the rule has an error action. **Fix**: Give every rule an errorAction to S3 or a second SQS queue, and alarm on the rule's Failure metric. - **Gotcha**: Two devices with the same client ID can't be connected at once: each new connection kicks the other off, and they fight in a loop. **Fix**: Use the thing name as the client ID and enforce it in the IoT policy (iot:Connect on client/${iot:Connection.Thing.ThingName}). - **Gotcha**: Nothing can subscribe to Basic Ingest topics. A dashboard added later can't see those readings live. **Fix**: Send bulk readings through Basic Ingest, and publish only what must be watched live to a normal topic. - **Gotcha**: MQTT connections aren't guaranteed to last: with certificates they can drop after minutes, and at most last a week or two. **Fix**: Use the AWS IoT Device SDKs, reconnect with backoff, and publish with QoS 1 so unacknowledged readings are resent. - **Gotcha**: Lifecycle events (connected, disconnected) can arrive twice or out of order, so a brief reconnect can look like a device going offline. **Fix**: Treat a disconnect as a reason to check, not an alert: confirm with the last heartbeat or reading, after a grace period, before raising anything. ## AWS Lambda [AWS Lambda](https://cloudlesslabs.com/stack/aws-lambda/index.md): My default. - **Gotcha**: A function idle for weeks goes Inactive, and the first calls fail while it reactivates. **Fix**: Ping it on a schedule, use provisioned concurrency for critical paths, or move bursty HTTP endpoints to Workers. - **Gotcha**: Scaling is capped: 1,000 concurrent executions per account by default, and each function adds at most 1,000 instances every 10 seconds. **Fix**: The limit is requests in flight, not per second (2,000 req/s at 100 ms is only 200). Raise the quota before launches, and put Workers in front of known spikes. - **Gotcha**: That quota is one pool shared by every function in the account and Region. A backlog in one function (a reconnect storm, a reprocessing job) can throttle another, including the one that matters most. New accounts start below 1,000. **Fix**: Raise the quota through Service Quotas (AWS goes to tens of thousands), check the real value with aws lambda get-account-settings, and give critical functions reserved concurrency: capacity no other function can take, and a cap. Lambda keeps 100 unreserved, so a 1,000 quota lets you reserve at most 900. - **Gotcha**: SnapStart doesn't help Node.js. As of October 2026 it only supports Java, Python, .NET and container images. **Fix**: For Node.js zip functions, keep the bundle small, initialise lazily, and use provisioned concurrency where latency matters. - **Gotcha**: Log-level filtering (drop DEBUG and INFO before you pay for them) only works with the JSON log format. The default Text format logs everything. **Fix**: Set the log format to JSON and the application log level to WARN in production. - **Gotcha**: Response streaming isn't available through a function URL when the function is in a VPC. **Fix**: Keep streaming endpoints out of the VPC, or stream from Workers or Lambda Web Functions. ## AWS Organizations + IAM Identity Center [AWS Organizations + IAM Identity Center](https://cloudlesslabs.com/stack/aws-organizations-identity-center/index.md): My default. - **Gotcha**: Long-lived IAM user keys scattered across laptops. **Fix**: SSO through Identity Center for every human; no static keys. - **Gotcha**: SCPs don't apply to anyone in the management account. Whatever runs there escapes every guardrail you write. **Fix**: Keep the management account empty: Organizations, Identity Center and billing only. Workloads go in member accounts. - **Gotcha**: Every member account still has a root user with a password, the strongest credential in the account. **Fix**: Turn on centralised root access in IAM and remove root credentials from member accounts. New accounts are created without them. - **Gotcha**: A new organization can hold 10 accounts by default, and a closed account keeps counting until it's permanently closed, 90 days later. **Fix**: Request the quota increase from the management account before the account plan needs it. - **Gotcha**: An SCP attached at the root that denies too much locks every member account out at once. **Fix**: Test it on a sandbox OU first, move accounts in a few at a time, and keep SCPs in Pulumi so a bad one is one revert away. ## AWS Secrets Manager [AWS Secrets Manager](https://cloudlesslabs.com/stack/aws-secrets-manager/index.md): My default. - **Gotcha**: After a rotation, ECS tasks and long-running processes keep using the old password: ECS injects secrets only when a container starts. **Fix**: Use the alternating-users rotation strategy for app users, retry once with a fresh read on an authentication failure, and redeploy from the EventBridge event Secrets Manager sends on every update. - **Gotcha**: A secret encrypted with the default aws/secretsmanager key can't be shared with another account, and switching keys later is painful. **Fix**: Create every secret on a customer managed KMS key from day one. - **Gotcha**: Reading the secret on every Lambda invocation multiplies API calls by concurrency: cost, and throttling at scale. **Fix**: Use the Parameters and Secrets Lambda Extension (cache up to 300 seconds) or Powertools with a cache TTL, or BatchGetSecretValue for many secrets at once. - **Gotcha**: Deleted secrets wait 7 to 30 days before they're gone, and the name can't be reused meanwhile, so tearing down and recreating a stack fails. **Fix**: Use a recovery window of 0 for development stacks only, and keep the default window in production. - **Gotcha**: The value ends up in plaintext anyway: in a Lambda environment variable, a CloudFormation template or a commit. **Fix**: Pass references, not values (valueFrom in ECS, {{resolve:secretsmanager:...}} in CloudFormation), and read at runtime. ## AWS Step Functions [AWS Step Functions](https://cloudlesslabs.com/stack/aws-step-functions/index.md): Never. - **Gotcha**: Each state's input and output is capped at 256 KB, and that limit can't be raised. **Fix**: Pass S3 keys or database IDs between states, never the data itself. - **Gotcha**: A Standard execution fails when its history reaches 25,000 events. Long loops hit it. **Fix**: Use a Distributed Map, or start a child execution per batch instead of looping inside one execution. - **Gotcha**: Every retry is billed as another state transition, so a flaky step quietly multiplies the bill. **Fix**: Count retries in the estimate, and cap MaxAttempts on each Retry block. - **Gotcha**: Express workflows are at least once when started asynchronously and stop at 5 minutes, so a step can run twice. **Fix**: Use Standard for anything that must not repeat (payments), or make every Express step idempotent. ## Better Auth [Better Auth](https://cloudlesslabs.com/stack/better-auth/index.md): My default. - **Gotcha**: There's no hosted login page. Better Auth gives you the endpoints and a client SDK, not the screens. **Fix**: Build sign-in, sign-up and reset-password pages in your app (a day with Claude Code), or use a ready-made component kit if the front end is React. They match your design, which Cognito's pages never quite do. - **Gotcha**: It doesn't send email. Password reset, verification and magic links produce a link and wait for you. **Fix**: Pass it to Resend in a few lines (sendResetPassword, sendVerificationEmail, sendMagicLink). - **Gotcha**: 27 security advisories in 2026 (3 critical), many in the young SSO and SCIM plugins. Other open-source auth projects had similar years. **Fix**: Pin exact versions, let Renovate or Dependabot open update PRs, and run a weekly scheduled agent that checks the repo's security advisories and opens the upgrade. Enable only the plugins you use. - **Gotcha**: Letting users register their own SSO providers caused most of the 2026 SSO advisories (account takeover through unproven domains). **Fix**: Register SSO providers only from your own server code, require DNS domain verification, turn on requireTimestamps for SAML, and keep implicit account linking off. - **Gotcha**: Its JWT plugin signs EdDSA by default, which API Gateway JWT authorizers and IAM don't accept. **Fix**: Configure RS256. Then Workers, API Gateway and Cognito identity pools all accept its tokens. - **Gotcha**: Google Workspace can't push users to custom apps over SCIM. **Fix**: Google sign-in works; for Google directory sync, use another tool (Ory Polis or WorkOS) or invite users by domain. ## Claude Code [Claude Code](https://cloudlesslabs.com/stack/claude-code/index.md): My default. - **Gotcha**: On Free, Pro and Max, Anthropic trains on your sessions when the privacy setting is on, and keeps that data for five years (30 days when it's off). Team, Enterprise, API and Bedrock usage isn't used for training. **Fix**: Put client work on a Team or Enterprise plan, a Console API key or Bedrock. On a personal plan, switch training off in claude.ai privacy settings first. - **Gotcha**: If ANTHROPIC_API_KEY is set in your shell, Claude Code uses it once approved, and always uses it in headless (-p) mode. Your subscription goes unused and the API bills you per token. **Fix**: Unset the variable unless you mean it, and check /status to see which credential is active. Use CLAUDE_CONFIG_DIR to keep work and personal accounts apart. - **Gotcha**: Session transcripts are stored in plain text under ~/.claude/projects for 30 days, including any secret the agent read along the way. **Fix**: Keep secrets out of the repo, add deny rules for files like .env, and shorten cleanupPeriodDays on shared or client machines. - **Gotcha**: Headless (-p) sessions skip the workspace trust and .mcp.json approval prompts, so a cloned repo's settings, hooks and MCP servers can run without anyone being asked. **Fix**: Run headless jobs only on repos you trust, inside a dev container or VM, with sandboxing on (/sandbox) for network and filesystem isolation. - **Gotcha**: Auto mode is now the starting permission mode in the terminal and VS Code: a classifier model, not you, approves most actions. Fetched pages, issues and MCP tool results can carry prompt injection, and the agent may act on it. **Fix**: On client systems, start in Manual mode or set deny rules for anything destructive, use trusted MCP servers only, deny curl and wget unless needed, and use a VM for anything that browses the web unattended. ## Cloudflare AI Gateway [Cloudflare AI Gateway](https://cloudlesslabs.com/stack/cloudflare-ai-gateway/index.md): Watching. - **Gotcha**: Logging is on by default, and each log can hold the full prompt and response. For an app handling personal data, that's a copy of it in Cloudflare you didn't plan for. **Fix**: Turn payload logging off per gateway, or send cf-aig-collect-log-payload: false on those requests to keep tokens, cost and status without the text. - **Gotcha**: With authentication off, the gateway accepts requests from anyone who knows your account and gateway IDs. That matters as soon as it holds your provider keys: stored keys require an authenticated gateway. **Fix**: Turn authentication on (authentication: true in Pulumi) and send the gateway token only from your server or Worker, never from the browser. - **Gotcha**: Spend limits are a best-effort estimate from token counts and known model prices, enforced with a lag: a burst of concurrent requests can overshoot, and models without known pricing aren't tracked. **Fix**: Treat them as a guardrail, not a wallet. Keep a hard credit limit on the OpenRouter key too, and the per-user budget in your Worker. - **Gotcha**: Caching is off by default and exact-match only. A timestamp or user ID inside the prompt makes every request unique, and two identical requests sent at once can both miss. **Fix**: Keep the cacheable part of the prompt identical, set cf-aig-cache-ttl per request, or set cf-aig-cache-key to group requests yourself. - **Gotcha**: Dynamic routing, and the fallbacks it carries, only accepts the OpenAI chat-completions format. An Anthropic Messages request (Claude Code, the Anthropic SDK) gets a 400. **Fix**: Send Anthropic-format traffic to the /anthropic endpoint, which still gets logs, retries, rate and spend limits, and keep route-based fallbacks for OpenAI-format calls, such as OpenRouter's. - **Gotcha**: A gateway token (a Cloudflare API token with Run permission) can't be limited to one gateway: it reaches every gateway and every stored provider key in the account. **Fix**: Give each client its own Cloudflare account when their keys must be isolated, and keep the token on the server side only. - **Gotcha**: Unified Billing (Cloudflare's prepaid credits) is capped at 200 requests per 60 seconds per gateway, and returns 429 above that. **Fix**: Use your own provider key (BYOK) for production traffic: the cap doesn't apply to it. ## Cloudflare D1 [Cloudflare D1](https://cloudlesslabs.com/stack/cloudflare-d1/index.md): Skip. - **Gotcha**: Billing counts rows scanned. A filter on an unindexed column reads the whole table on every query, and so does the bill. **Fix**: Index the columns you filter and sort on, and check rows_read in each query's meta object (or the dashboard) before going live. - **Gotcha**: The 10 GB per database cap is hard: Cloudflare raises most D1 limits on request, but not that one. **Fix**: Decide on day one how the data splits (per tenant, per year), or use Postgres if one database will outgrow it. - **Gotcha**: A Time Travel restore overwrites the database in place, and you can't restore into a new copy. **Fix**: Keep the undo bookmark the restore returns, and export the database before restoring if you only need a few rows back. - **Gotcha**: On the Free plan, since September 2026, queries fail once the daily row limits are hit, until 00:00 UTC. **Fix**: Anything a client depends on goes on Workers Paid. ## Cloudflare Durable Objects [Cloudflare Durable Objects](https://cloudlesslabs.com/stack/cloudflare-durable-objects/index.md): Use when it fits. - **Gotcha**: Single-threaded isn't automatically transactional across an await. If you await an outside call (asking AWS whether a bidder is verified) between reading the price and writing the bid, other bids run in the gap. **Fix**: Do outside checks before the critical section. Keep read → check → write → broadcast synchronous (the SQLite API is synchronous, or use transactionSync). - **Gotcha**: Every deploy restarts all Durable Objects and drops every WebSocket. **Fix**: Clients must reconnect automatically. Don't deploy during a live event. - **Gotcha**: No published maximum of WebSockets per object. The real ceilings are one thread, 128 MB and about 1,000 incoming requests a second. **Fix**: Up to about 2,000 watchers connect directly. Above that, fan out through helper objects that each hold 1,000 to 2,000 sockets, and batch broadcasts every 50–100 ms. - **Gotcha**: Alarms can fire more than once. **Fix**: Make the alarm handler idempotent: check the object's state before acting. - **Gotcha**: An object is created near its first request (or where locationHint says) and never moves. **Fix**: Pass locationHint 'oc' on first access for Australian users. - **Gotcha**: 10 GB per object, 100 columns per table, 2 MB per row. **Fix**: Design the schema around one entity (this auction, this room), and export finished data to AWS. ## Cloudflare Hyperdrive [Cloudflare Hyperdrive](https://cloudlesslabs.com/stack/cloudflare-hyperdrive/index.md): Skip. - **Gotcha**: Query caching is on by default (60 seconds, plus 15 seconds stale), and writes don't invalidate it. A user saves a change and the next read shows the old value. **Fix**: Disable caching on the config for transactional apps (--caching-disabled, or caching.disabled in Pulumi). If some reads can be cached, bind a second, cache-disabled config for auth, permissions and reads after writes. - **Gotcha**: Pooling is in transaction mode: the connection goes back to the pool after each transaction, so session state set with SET is lost. **Fix**: Pass settings per query or inside the transaction, and use a separate direct client for the rare statement that needs a session. - **Gotcha**: Each config opens about 100 connections to the database at most on Workers Paid (a soft limit). A second config, such as a cache-disabled one, adds its own. **Fix**: Set originConnectionLimit for every config so the total stays under the database's max_connections, and ask Cloudflare to raise it if needed. - **Gotcha**: For Neon, Cloudflare's guide says to use the direct (non-pooled) connection string with pg or Postgres.js, not Neon's serverless driver. **Fix**: Untick connection pooling when copying the Neon string, and let Hyperdrive do the pooling. ## Cloudflare Queues [Cloudflare Queues](https://cloudlesslabs.com/stack/cloudflare-queues/index.md): Skip. - **Gotcha**: Without a dead-letter queue, failed messages are deleted. **Fix**: Always configure a DLQ. Name it in the consumer settings and Cloudflare creates it if it doesn't exist. Retries default to 3. - **Gotcha**: Delivery is at least once: a consumer can see the same message twice. **Fix**: Give each message an ID when you send it, and use it as a unique key or idempotency key downstream. - **Gotcha**: Each queue takes at most 5,000 messages a second; above that, send() throws Too Many Requests. **Fix**: Pack several small events into one message (stay under 64 KB, which is still one operation), or spread producers across several queues. - **Gotcha**: On the Free plan, retention is fixed at 24 hours. A consumer that's broken over a weekend loses messages. **Fix**: Use Workers Paid for anything that matters, and set retention explicitly (up to 14 days; the paid default is 4). - **Gotcha**: A queue's jurisdiction can only be set when it's created. **Fix**: Decide residency before creating the queue. Changing it later means a new queue and a cut-over. ## Cloudflare R2 [Cloudflare R2](https://cloudlesslabs.com/stack/cloudflare-r2/index.md): Use when it fits. - **Gotcha**: The r2.dev public URL is for development only. It's rate-limited (roughly hundreds of requests a second, then 429s), may throttle bandwidth, and has no cache, WAF or access rules. **Fix**: Serve production traffic from a custom domain on a Cloudflare zone, with cache rules in front of the bucket. - **Gotcha**: WAF rules or Access on the custom domain don't protect the bucket if r2.dev is still on: the same files stay public at the r2.dev address. **Fix**: Turn off the r2.dev subdomain as soon as the custom domain is live. - **Gotcha**: R2 accepts one write per second to the same key. Faster rewrites of one object (a counter, a status file) get HTTP 429. **Fix**: Don't use R2 as a database. Keep changing state in a Durable Object or Postgres, and write immutable, uniquely named objects to R2. - **Gotcha**: There's no versioning: an overwrite or a delete is final. **Fix**: Use bucket locks to protect prefixes that must not change, or keep the system of record in S3 with versioning and treat R2 as a delivery copy. - **Gotcha**: A location hint (oc for Oceania) is best effort and applies only when the bucket is first created; a jurisdiction can never be changed afterwards. **Fix**: Set the hint in the Pulumi resource on creation. If the data must stay in Australia, use S3 in Sydney instead. ## Cloudflare Sandbox [Cloudflare Sandbox](https://cloudlesslabs.com/stack/cloudflare-sandbox/index.md): Use when it fits. - **Gotcha**: Untrusted code can still try to reach your APIs or leak secrets through the network. **Fix**: Never put secrets inside the sandbox. Route its traffic through an outbound Worker, the egress proxy that injects credentials per destination so the code never sees them, and allow only the destinations it needs. - **Gotcha**: Costs grow with every session left running. **Fix**: Give every session a timeout and destroy it when the user leaves; back up the workspace and restore it if they come back. ## Cloudflare static hosting [Cloudflare static hosting](https://cloudlesslabs.com/stack/cloudflare-static-hosting/index.md): My default. - **Gotcha**: An SPA's deep links (/dashboard/settings) return 404 on a fresh load, because there's no file at that path. **Fix**: Set not_found_handling = "single-page-application" in the Wrangler config. Unmatched paths then get /index.html with a 200. - **Gotcha**: Pages preview deployments are public by default. Every branch build of a client's unreleased site has a guessable URL. **Fix**: Turn on the access policy for previews (Settings, General, Enable access policy), or Cloudflare Access on the preview hostnames for Workers. - **Gotcha**: With a Worker script and a compatibility date from April 2025 on, browser navigations skip the Worker and get the HTML. Opening /api/report in a tab returns the SPA, while fetch() still reaches the API. **Fix**: List the API routes in run_worker_first (for example ["/api/*"]) so they always hit the Worker. - **Gotcha**: Limits per version: 20,000 files on Workers Free (100,000 on Paid), and 25 MiB per file. Pages also caps builds at 500 a month on the free plan, one at a time, 20 minutes each. **Fix**: Keep video and downloads in R2, prune generated files, and build in GitHub Actions if the free build quota or timeout gets tight. ## Cloudflare Tunnel (cloudflared) [Cloudflare Tunnel (cloudflared)](https://cloudlesslabs.com/stack/cloudflare-tunnel/index.md): My default. - **Gotcha**: A tunnel doesn't close anything. If the service still listens on 0.0.0.0, it's reachable on the box's public IP, around Cloudflare and around your Access login. **Fix**: Bind every service to 127.0.0.1, drop all inbound traffic in the firewall (keep your own SSH path first), and allow outbound port 7844 on TCP and UDP for cloudflared. - **Gotcha**: Cloudflare waits 125 seconds for the origin to answer, then returns error 524. Only Enterprise plans can raise it. **Fix**: Return 202 straight away and let the client poll for the result, or stream the response so bytes start flowing early. - **Gotcha**: Request bodies are capped at 100 MB on Free and Pro (200 MB on Business). Bigger uploads get a 413. **Fix**: Upload big files straight to R2 or S3 with a presigned URL; the box only receives the key. - **Gotcha**: Cloudflare only supports cloudflared releases from the past year; older ones can break without notice. **Fix**: Install it from Cloudflare's package repository and let the system's updates keep it current, or bump it on a schedule. - **Gotcha**: If the tunnel stops, its DNS record stays, and visitors get Cloudflare error 1016 instead of the site. **Fix**: Run cloudflared as a system service that restarts on failure, add a second replica on another machine if the service matters, and alert on the tunnel's health. ## Cloudflare Turnstile [Cloudflare Turnstile](https://cloudlesslabs.com/stack/cloudflare-turnstile/index.md): My default. - **Gotcha**: The widget alone doesn't stop bots: they post straight to your endpoint and never load it. **Fix**: Check every token on the server, with your secret, at https://challenges.cloudflare.com/turnstile/v0/siteverify, and reject the request unless it succeeds. - **Gotcha**: A token can be checked only once and expires after 5 minutes, so a slow form or a retry fails. **Fix**: Verify once, at submission; if the form errors, reset the widget for a new token before the user resubmits. - **Gotcha**: Some bots still get through: Turnstile tells you a browser was there, not that a human meant it. **Fix**: Add a hidden honeypot field and drop any submission that fills it. ## Cloudflare Workers [Cloudflare Workers](https://cloudlesslabs.com/stack/cloudflare-workers/index.md): My default. - **Gotcha**: It's not full Node.js. Most pure-JavaScript npm packages work, but there are no native modules and child_process, worker_threads, cluster and vm are stubs. **Fix**: Treat Workers as the layer in front of your backend. If a Worker grows into business logic that needs these, move it to Lambda Web Functions. - **Gotcha**: Memory is fixed at 128 MB per isolate. **Fix**: Stream instead of buffering. Anything heavy goes to Lambda. - **Gotcha**: Calling AWS from a Worker means storing access keys: there's no IAM role. **Fix**: Use narrowly scoped keys in Workers secrets, or call an AWS endpoint (function URL with signed requests) instead of AWS APIs directly. - **Gotcha**: Cron triggers are UTC only. **Fix**: Convert schedules to UTC, or schedule in AWS with EventBridge Scheduler. - **Gotcha**: Workers Logs bills per log event until 30 November 2026 (20M included, then $0.60 per million). Five console.log lines per request multiplies the bill by five. **Fix**: Log one structured object per request. From 1 December 2026 billing moves to $0.25/GB, cheaper than CloudWatch in Sydney ($0.67/GB). ## Cloudflare Workers AI [Cloudflare Workers AI](https://cloudlesslabs.com/stack/cloudflare-workers-ai/index.md): Skip. - **Gotcha**: The paid-only frontier models are limited to 20 requests a minute per account and model. At steady traffic that's under 900,000 requests a month, and real traffic is never steady. **Fix**: Don't put those models behind a production feature. Use OpenRouter, or load prepaid AI Gateway credits and switch the gateway to Unified Billing for 50 a minute, or ask Cloudflare for a custom limit. - **Gotcha**: The AI binding always runs in your Cloudflare account, even under wrangler dev. Local development and test suites burn real neurons. **Fix**: Develop on a small model inside the free daily allocation, and mock the binding in unit tests. - **Gotcha**: Cached-input pricing exists only on some models (DeepSeek, Kimi, GLM). On gpt-oss-120b, Gemma 4 and Llama 4 Scout, a long fixed system prompt is billed in full on every call. **Fix**: For long fixed prompts, pick a model with cached-input pricing, or keep the prompt short. ## CloudFront Functions [CloudFront Functions](https://cloudlesslabs.com/stack/aws-cloudfront-functions/index.md): Use when it fits. - **Gotcha**: The runtime is a restricted JavaScript: no network, no file system, no environment variables, no timers, no request body, and the code is capped at 10 KB with 2 MB of memory. **Fix**: Keep it to string work on the URL and headers. Values that change, such as a redirect table, go in KeyValueStore (runtime 2.0) rather than in the code. - **Gotcha**: A function runs on every request of the behaviour it's attached to, assets included. **Fix**: Attach it to the default behaviour for pages only, and give /_next/static/* and other asset paths their own behaviours without it. - **Gotcha**: A function has a DEVELOPMENT and a LIVE stage: saving new code in the console, or deploying it with publish set to false, leaves the distribution running the old LIVE version. **Fix**: Keep publish: true in code (the default), and test the new code with aws cloudfront test-function before it goes live. - **Gotcha**: On a flat-rate plan, a function, and its KeyValueStore, can't be shared with another distribution. **Fix**: Create one function per distribution in code, from the same source. ## Cost Explorer (with Claude Code) [Cost Explorer (with Claude Code)](https://cloudlesslabs.com/stack/aws-cost-explorer/index.md): My default. - **Gotcha**: Operation names can mislead: CloudWatch log ingestion shows as 'PutLogEvents' but is billed per GB. **Fix**: Group by usage type as well as operation. - **Gotcha**: The default view shows upfront Reserved Instance and Savings Plan fees as a spike on the day they're charged, so one month looks terrible and the next looks cheap. **Fix**: Switch to amortized costs for any trend or per-team comparison. - **Gotcha**: Costs only group by a tag after the tag key is activated as a cost allocation tag, which can take up to 24 hours to appear and another 24 to activate. **Fix**: Activate the tag keys on day one, in code (costexplorer.CostAllocationTag), before the first bill you'll need to split. - **Gotcha**: AWSBillingReadOnlyAccess lets an agent read costs but not forecasts, anomalies or Savings Plans recommendations. And AWS's Billing and Cost Management MCP server isn't strictly read-only: some of its tools start analyses and write to S3 and Athena. **Fix**: Give the agent its own Identity Center permission set with ce:Get*, ce:Describe* and ce:List*, and nothing that can buy or change. - **Gotcha**: Cost Anomaly Detection runs about three times a day on data that can lag 24 hours, and needs 10 days of history before it flags a new service. **Fix**: Treat it as a daily safety net, not an alarm. Hard limits in the architecture stop the bleeding. ## Dashlane [Dashlane](https://cloudlesslabs.com/stack/dashlane/index.md): Use when it fits. - **Gotcha**: Leaving is harder than joining: the CSV export carries passwords, but passkeys can't be exported by CSV or backed up in a DASH file. **Fix**: Move passkeys with Credential Exchange, which Dashlane supports with 1Password, Bitwarden, Apple Passwords and Google Password Manager. Otherwise, register a new passkey on each site after the move. Delete the CSV as soon as the import is done. - **Gotcha**: If a business plan isn't paid, a 30-day grace period starts. After it, members can't log in at all until an admin pays. **Fix**: Pay annually from an account nobody forgets, keep two admins on billing, and keep an encrypted export offline for emergencies. - **Gotcha**: A lapsed personal plan becomes export-only: you can log in, but you can't view, copy or edit your passwords. Every free account went this way when Free ended on 16 September 2025, work logins included. **Fix**: Put every work credential in the business plan, never in personal accounts. ## DataForSEO [DataForSEO](https://cloudlesslabs.com/stack/dataforseo/index.md): My default. - **Gotcha**: It doesn't use an API key: it signs in with the API login and password from the API Access dashboard. **Fix**: Keep them in a gitignored .env or your secrets manager, never in the chat. seo-pilot sets them up this way. - **Gotcha**: Pay as you go also means an agent in a loop can spend the balance. **Fix**: Set a daily spending limit for the account, or per API, in the dashboard, and top up by hand rather than automatically. - **Gotcha**: The same request costs more in Live mode than in the Standard queue, which is cheaper but slower. **Fix**: Use the Standard queue for batch audits and keyword lists; keep Live for answers you need now. - **Gotcha**: The MCP server exposes dozens of tools, and some clients cap them (Cursor allows 40 active). **Fix**: Enable only the modules you need, or call the API through a skill: seo-pilot uses its dataforseo-web-api skill instead of the MCP server. ## DuckDB [DuckDB](https://cloudlesslabs.com/stack/duckdb/index.md): Use when it fits. - **Gotcha**: Lambda has no HOME directory, so INSTALL, LOAD and CREATE SECRET fail with "Can't find the home directory". **Fix**: Run SET home_directory='/tmp' first. - **Gotcha**: The S3, AWS and Iceberg extensions are downloaded at runtime unless bundled: about 100 MB on every cold start. **Fix**: Bundle them in the layer and SET extension_directory to it. - **Gotcha**: DuckDB SQL isn't Athena's (Trino) SQL. Date, array and approximate functions differ. **Fix**: Test shared queries on both engines, or keep each query on one engine. - **Gotcha**: The layer zip is over 50 MB. **Fix**: Upload the layer through S3, or use a container image. - **Gotcha**: DuckDB 2.0.0 LTS (21 October 2026) writes database files that 1.x can't read, and 1.4 LTS reaches end of support on 17 November 2026. **Fix**: Move to 2.0 (supported until October 2027) in one step: upgrade every reader before anything writes a 2.0 file. Parquet and Iceberg files in the lake aren't affected. ## EC2 [EC2](https://cloudlesslabs.com/stack/aws-ec2/index.md): Use when it fits. - **Gotcha**: Burstable t4g instances sustain only 20 to 40% of each vCPU before their credits run out. **Fix**: Use them for light, idle-heavy loads; size steady work on c7g or m7g. - **Gotcha**: Every instance is an operating system to patch, an image to rebuild and an autoscaling group to keep healthy. **Fix**: Use Fargate unless you need what only EC2 has, and automate images and patching from day one when you do. ## Fargate scheduled tasks [Fargate scheduled tasks](https://cloudlesslabs.com/stack/aws-fargate-scheduled-tasks/index.md): Use when it fits. - **Gotcha**: Fargate Spot tasks can be stopped with two minutes' notice. **Fix**: Use Spot only for jobs that checkpoint or can safely restart. Handle SIGTERM in the container and set stopTimeout to 120 seconds so the job can save its progress. - **Gotcha**: Fargate Spot capacity can be unavailable at busy times, and Fargate doesn't fall back to on-demand. **Fix**: Run jobs that must finish by a deadline on regular Fargate, and keep Spot for work that can start late. - **Gotcha**: The task must reach ECR to pull its image. In a private subnet that means a NAT gateway, which bills every hour of the month for a job that runs two. **Fix**: Run the task in a public subnet with a public IP and a security group that allows no inbound traffic, or add ECR and S3 VPC endpoints if it must stay private. - **Gotcha**: The schedule only knows that RunTask was accepted. If the container crashes or exits with an error, nothing retries and nobody is told. **Fix**: Add an EventBridge rule on ECS Task State Change events (STOPPED with a non-zero exit code) that alerts you, and make the job safe to run again. - **Gotcha**: A schedule set in local time is skipped on the night daylight saving starts if it falls in the missing hour (2–3 am in Sydney). **Fix**: Schedule nightly jobs outside 2–3 am local time, or in UTC. ## Fargate services (always on) [Fargate services (always on)](https://cloudlesslabs.com/stack/aws-fargate-service/index.md): Use when it fits. - **Gotcha**: Target-tracking scaling reacts on one-minute metrics with a cooldown, then new tasks must start: a sudden ten-minute peak is mostly over before the capacity arrives. **Fix**: Keep the spiky part on Lambda, or use scheduled scaling when the peak's time is known. - **Gotcha**: Tasks in private subnets need a NAT gateway to reach ECR, Neon or any API: about $43 a month before data, for every zone. **Fix**: Run tasks in public subnets with public IPs and a security group that only accepts the load balancer, or add VPC endpoints when they must stay private. - **Gotcha**: App Runner, the easy way to run a container service, is closed to new customers. **Fix**: Use ECS Express Mode: one call creates the Fargate service, the load balancer with TLS, health checks and scaling, at no extra charge. - **Gotcha**: Every Fargate service on the internet gets its own load balancer, a fixed monthly cost before its first request. **Fix**: Since 20 October 2026, ECS Gateway shares one managed load balancer across up to 25 services, free on top of the load balancer itself. It routes by host name, HTTPS on 443 only. CloudFormation has it (AWS::ECS::Gateway); Terraform doesn't yet. ## GitHub [GitHub](https://cloudlesslabs.com/stack/github/index.md): My default. - **Gotcha**: On the Free plan, private repositories get no protected branches, no required reviewers and no code owners. Anyone with write access, including an agent's token, can push straight to main. **Fix**: Move the organisation to Team as soon as there's more than one person or agent writing, and declare the branch rules in Pulumi. - **Gotcha**: Secret scanning push protection, the check that blocks a commit containing an API key, is on by default for public repos only. On private repos it needs GitHub Secret Protection, a paid add-on. **Fix**: Pay for it on client repos, or run a secrets scanner such as gitleaks in a pre-commit hook and in CI. Keep secrets out of the repo entirely: environment files stay git-ignored. - **Gotcha**: A classic personal access token given to an agent or a script reaches every repo you can reach, and often never expires. **Fix**: Use fine-grained tokens scoped to the repos the job needs, with an expiry, or a GitHub App for anything organisation-wide or long-lived. GitHub recommends both. ## GitHub Actions [GitHub Actions](https://cloudlesslabs.com/stack/github-actions/index.md): Use when it fits. - **Gotcha**: AWS access keys stored as repository secrets: long-lived keys that work from anywhere and outlive the people who created them. **Fix**: OIDC: GitHub's token is exchanged for a short-lived IAM role. Pin the trust policy's sub claim to the repository and branch (or environment), never a wildcard. - **Gotcha**: Third-party actions referenced by tag can be repointed. In March 2025 the tj-actions/changed-files tags were moved to a malicious commit that printed secrets into build logs, across thousands of repositories. **Fix**: Pin every third-party action to a full commit SHA, or adopt workflow lockfiles (public preview since October 2026). Keep the list of actions short. - **Gotcha**: pull_request_target runs with the repository's secrets and write access, even for pull requests from forks. Checking out the fork's code there hands it your secrets. **Fix**: Use pull_request for anything that runs contributed code. If you need pull_request_target, never check out or run the pull request's code in it. - **Gotcha**: Two merges close together start two pulumi up runs on the same stack. One of them fails on the stack lock. **Fix**: A concurrency group per stack with cancel-in-progress: false, so the second run waits instead of fighting. - **Gotcha**: Required reviewers and wait timers on environments only work on private repositories with GitHub Enterprise. On Free, Pro and Team they're public-only. **Fix**: On those plans, gate production with branch protection on main, a separate production workflow, and an OIDC trust policy that only accepts main. ## GitLab, Bitbucket and Atlassian [GitLab, Bitbucket and Atlassian](https://cloudlesslabs.com/stack/gitlab-bitbucket-atlassian/index.md): Never. - **Gotcha**: Atlassian is shutting down Data Center. New customers can't buy it since 30 March 2026, existing ones can buy and expand until 30 March 2028, and on 28 March 2029 the licences expire and Jira and Confluence go read-only. Bitbucket and Bamboo are the exception. **Fix**: Plan the exit now, not in 2028: code to GitHub, tickets to GitHub Issues or a minimal Trello board, docs to Markdown in the repo. - **Gotcha**: Moving off GitLab or Bitbucket: GitHub Enterprise Importer moves pull requests and history from GitLab and Bitbucket Server/Data Center, but only into GitHub Enterprise Cloud, and Bitbucket Cloud isn't a supported source. **Fix**: For code and full history on any GitHub plan, git clone --mirror then git push --mirror. Rebuild CI as GitHub Actions. Accept that old pull-request discussions stay behind, or pay for Enterprise Cloud for the move. - **Gotcha**: The free tiers are small: GitLab.com Free caps a private group at 5 users and 400 compute minutes a month; Bitbucket Free caps at 5 users and 50 build minutes, and a 2x runner burns minutes twice as fast. **Fix**: Don't size a team on the free tier. If the client is staying, budget the paid plan; if it's moving, move before the team grows. ## Google Cloud [Google Cloud](https://cloudlesslabs.com/stack/google-cloud/index.md): Skip. - **Gotcha**: A normal budget only sends alerts. It never stops usage or spending. **Fix**: Use spend-cap budgets where they exist (Cloud Run, Cloud Run functions, the Gemini API and Google's agent platform): they pause that service in that project until you lift the cap. Elsewhere, wire budget notifications through Pub/Sub to disable billing on the project, and set the budget below your real limit, because cost data arrives late. - **Gotcha**: BigQuery's default on-demand model bills every byte a query scans. On a table that isn't clustered, LIMIT doesn't reduce what's read, so SELECT * … LIMIT 10 on a big table costs the full scan. **Fix**: Partition and cluster large tables, set maximum bytes billed on queries (a query over the limit fails before it costs anything), and add custom daily query quotas per user or project. - **Gotcha**: Cloud Run's default request-based billing gives the container CPU only while a request is in flight. Work left running after the response (an async job, a fire-and-forget call) stalls. **Fix**: Switch the service to instance-based billing for short background tasks, or hand longer work to Cloud Tasks. ## GraphQL and AppSync [GraphQL and AppSync](https://cloudlesslabs.com/stack/aws-appsync/index.md): Never. - **Gotcha**: Subscriptions fire only on mutations that go through AppSync. A row written by a Lambda, a queue consumer or a migration never reaches subscribers. **Fix**: Add a mutation on a NONE data source whose only job is to announce the change, and call it from the writer with IAM auth. Or use AppSync Events, which is plain publish and subscribe. - **Gotcha**: APPSYNC_JS resolvers aren't full JavaScript: no try/catch, no throw, no while loops, no promises or async, no network calls, and a 32 KB code limit. **Fix**: Keep resolvers to mapping only, and put real logic in a Lambda data source. At that point ask why the Lambda isn't the API. - **Gotcha**: Request execution time is capped at 30 seconds, and the cap can't be raised. **Fix**: Start long work asynchronously (SQS, a durable function) and return an ID the client can poll or subscribe to. - **Gotcha**: Subscription messages are capped at 240 KB. **Fix**: Push a small change notice and let the client fetch the full record over HTTP. ## HappySkills [HappySkills](https://cloudlesslabs.com/stack/happyskills/index.md): My default. - **Gotcha**: Installs fail without a clear error on Node.js older than 22. **Fix**: Upgrade Node.js to 22 or later before running npx happyskills setup. - **Gotcha**: A newly installed skill doesn't appear: agents read skills when a session starts. **Fix**: Restart the agent session after installing or updating skills. - **Gotcha**: A SKILL.md over 500 lines, or a forbidden character in its frontmatter, makes the skill silently invisible to the agent. **Fix**: Ask "Use HappySkills to audit my skill": it flags both before you publish, and validate refuses to ship a broken one. - **Gotcha**: Without the prefix, the agent may answer from memory instead of running HappySkills. **Fix**: Start requests with "Use HappySkills to …", or use the slash commands in Claude Code (/happyskills-search and the others). - **Gotcha**: update refuses to touch a skill you've edited locally. **Fix**: That's on purpose: use pull, which does a three-way merge from the version you started from, or --force to discard your edits. ## Hostinger VPS [Hostinger VPS](https://cloudlesslabs.com/stack/hostinger-vps/index.md): My default. - **Gotcha**: The CPU is credit-based. The baseline is 60% of the plan's CPU; a full credit balance lasts about three hours at 100%, then the box is capped at 30% (40% on KVM 1) until credits rebuild to 20%. **Fix**: Keep sustained work under the baseline, cap worker counts for tests and builds, and send long CPU-heavy jobs to a Fargate scheduled task that runs only while it's needed. - **Gotcha**: The headline price is an introductory rate, paid upfront for the whole term. Renewal is roughly double (KVM 2: $8.79 intro, $14.99 at renewal). **Fix**: Budget on the renewal price, and compare that number, not the intro one, against the alternatives. - **Gotcha**: Nothing on the box survives a rebuild unless it's written down. A VPS configured by hand is a snowflake. **Fix**: Provision the VPS with the Terraform provider and keep every package, service and timer in a Git repo with a sync script that rebuilds the box from scratch. - **Gotcha**: A public IP on an always-on box is a target from day one. **Fix**: Open no inbound ports beyond SSH (or none at all): publish services through Cloudflare Tunnel and reach the box over Tailscale. ## Jira, Asana, Monday.com and heavy project-management software [Jira, Asana, Monday.com and heavy project-management software](https://cloudlesslabs.com/stack/project-management-suites/index.md): Never. - **Gotcha**: Jira Software, Jira Service Management and Confluence Data Center reach end of life on 28 March 2029 and go read-only. New customers can't buy Data Center since 30 March 2026; existing ones can expand until 30 March 2028. **Fix**: Treat the forced move as a decision, not a migration project. Ask what you'd lose without the suite. If you keep it, move to Atlassian Cloud early; if not, export the history and move the work to GitHub and a simple board. - **Gotcha**: Atlassian's Rovo MCP server lets agents read and update Jira and Confluence, but every call draws on the organisation's shared pool of Rovo credits. **Fix**: Use it for reads and status updates, budget the credits, and keep the specs agents work from in the repo, not behind a metered connector. ## Lambda container images [Lambda container images](https://cloudlesslabs.com/stack/aws-lambda-container-images/index.md): Skip. - **Gotcha**: Cognito gives trigger functions 5 seconds, not configurable. A cold container image, or one gone Inactive after a few quiet weeks, can blow through it; after its retries, the sign-up or sign-in fails. **Fix**: Use zip functions for every Cognito trigger, keep them small, and push anything slow to an async job. - **Gotcha**: After a few weeks with no invocations, Lambda reclaims the optimised copy of the image and the function goes Inactive. The next call is rejected while Lambda prepares it again. **Fix**: Use zip for anything rarely called. If an image function must stay, invoke it on a schedule so it never goes inactive. - **Gotcha**: Lambda pins the image's digest at deploy time and re-reads it from ECR from time to time. When a rebuild pushes the same tag again, the running digest loses its tag, and an ECR lifecycle rule that expires untagged images deletes it. The function goes into a failed state and every call errors until you push the image back and redeploy. Nothing warns you. **Fix**: Give every image a unique tag, turn on tag immutability, keep lifecycle rules away from any digest a function, version or alias may still run, and alert on the function's state. - **Gotcha**: Nobody patches an image for you. AWS stops updating its base images when the runtime is deprecated, and your function only gets fixes when you rebuild and redeploy. **Fix**: Rebuild and redeploy images on a schedule, or use a zip function on a managed runtime, which Lambda patches for you. - **Gotcha**: Security fixes show the other side of escaping deprecations. When Node.js fixed its Brotli decoder (CVE-2026-111497, October 2026), zip functions on the managed runtime with automatic updates got the fix from AWS without a deploy; an image only gets it when you rebuild and redeploy. **Fix**: Know which of your functions are images, and rebuild them the day a runtime security release ships. - **Gotcha**: Lambda doesn't accept multi-architecture images: the image must target exactly one architecture, matching the function. **Fix**: Build with an explicit platform (linux/arm64 or linux/amd64) and set the same architecture on the function. - **Gotcha**: Images can't use Lambda layers, and the ECR repository must be in the function's region. **Fix**: Bake shared code into the base image, and create the repository in the same Pulumi stack and region as the function. ## Lambda durable functions [Lambda durable functions](https://cloudlesslabs.com/stack/aws-lambda-durable-functions/index.md): Watching. - **Gotcha**: Durable execution can only be switched on when the function is created. You can't enable it on an existing function. **Fix**: Plan it as a new function with its own name, and move the trigger over once it's proven. - **Gotcha**: Replay re-runs all code outside steps from the top. Date.now(), random IDs or reads done outside a step change between replays. **Fix**: Put everything non-deterministic inside a step, including idempotency keys, so its result is saved and reused. - **Gotcha**: Steps are at least once per attempt by default: if the environment dies mid-step, the step runs again. **Fix**: Pass an idempotency key to external APIs. For a charge that must not repeat, use AtMostOncePerRetry semantics with no retries. - **Gotcha**: Renaming a step or changing what it does breaks executions already in flight, which replay against the old step names. **Fix**: Invoke through a published version or alias (required anyway), keep old versions until in-flight runs finish, and add new steps instead of renaming. - **Gotcha**: When an asynchronously invoked durable execution fails, Lambda doesn't retry it, and Lambda destinations aren't supported. **Fix**: Configure a DLQ on the function and an EventBridge rule that alerts on FAILED, STOPPED and TIMED_OUT executions. ## Lambda function URLs [Lambda function URLs](https://cloudlesslabs.com/stack/aws-lambda-function-urls/index.md): My default. - **Gotcha**: Response streaming doesn't work through a function URL for a function inside a VPC. **Fix**: Keep streaming functions outside the VPC, or use Workers or Lambda Web Functions. - **Gotcha**: Since October 2025, a new function URL needs both lambda:InvokeFunctionUrl and lambda:InvokeFunction in its resource policy. IaC that grants only the first returns 403, even with auth type NONE. **Fix**: Grant both permissions, and scope the second with the lambda:InvokedViaFunctionUrl condition so the function can't be invoked any other way. - **Gotcha**: There's no custom domain: the URL is a random ID on `.lambda-url..on.aws`, and it changes if you delete and recreate it. **Fix**: Put CloudFront or a Cloudflare Worker route on your domain in front, and never hand the raw URL to a client. - **Gotcha**: Behind CloudFront with origin access control (auth type AWS_IAM), every POST or PUT must carry a SHA-256 hash of the body in x-amz-content-sha256. Browsers don't send it. **Fix**: Compute the hash in the client, or keep auth type NONE and check a secret header that only CloudFront or your Worker adds. - **Gotcha**: The only throttle is reserved concurrency: above it, or above 10 requests a second per unit of it, callers get 429. There are no per-client limits. **Fix**: Set reserved concurrency as a ceiling that protects the database, and do per-client rate limiting in a Worker or with WAF on CloudFront. ## Lambda Managed Instances [Lambda Managed Instances](https://cloudlesslabs.com/stack/aws-lambda-managed-instances/index.md): Watching. - **Gotcha**: The function needs at least 2 GB and 1 vCPU, and the instances run in your VPC. **Fix**: Use public subnets where you can: a NAT gateway adds a fixed monthly cost. - **Gotcha**: Many requests share one environment, so module-level state is shared between them. **Fix**: Keep handlers free of per-request globals before moving a function onto it. ## Lambda Web Functions [Lambda Web Functions](https://cloudlesslabs.com/stack/aws-lambda-web-functions/index.md): Watching. - **Gotcha**: Up to 128 requests share one process. Lambda code that keeps the current user or tenant in a module-level variable becomes a data leak between users. **Fix**: Audit global variables before migrating. Keep per-request data in AsyncLocalStorage or pass it explicitly. - **Gotcha**: A brand-new Web Function starts with a conservative per-environment concurrency limit and learns the right one from traffic. Your first big burst uses more environments and cold-starts more. **Fix**: Load-test before launch and judge by the second run. Lambda remembers what it learned, even after scaling to zero. - **Gotcha**: No minimum or reserved environments, and AWS doesn't publish how fast new ones are added. A burst faster than the spare headroom waits on cold starts, and over the limits requests get HTTP 429, not a queue. **Fix**: Clients retry with backoff and jitter. For known spikes, put Workers in front. - **Gotcha**: Default limits per region: 10,000 req/s per endpoint, 100,000 per account, 2,000 vCPUs (1,000 environments). **Fix**: Request increases ahead of growth, and set rateLimit on endpoints to protect Postgres. - **Gotcha**: Node.js 24 on Arm64 only, 2 GB and 2 vCPU per environment. **Fix**: Keep anything bigger on classic Lambda. ## Microsoft Azure, Windows and the Microsoft stack [Microsoft Azure, Windows and the Microsoft stack](https://cloudlesslabs.com/stack/microsoft-azure/index.md): Never. - **Gotcha**: When Entra ID provisions users into AWS IAM Identity Center over SCIM, it doesn't read nested groups: only direct members of a group assigned to the app are synced. And an attribute removed from a user in Entra isn't removed in Identity Center. **Fix**: Assign flat groups directly to the AWS IAM Identity Center enterprise app in Entra, and audit the synced attributes your permission sets depend on. - **Gotcha**: Azure budgets never stop spending: they send alerts, and cost data arrives 8 to 24 hours late. **Fix**: Set thresholds well below the real limit and attach an action group that runs automation (a function or webhook that scales down or shuts off) when one is crossed. - **Gotcha**: Azure Functions on the Linux Consumption plan retire on 30 September 2028 and get no new language versions. Apps still on the v3 runtime there stopped running on 30 September 2026. **Fix**: Migrate to the Flex Consumption plan: it creates a new app alongside the old one. Run az functionapp flex-migration list to see which apps are eligible and what blocks the others. - **Gotcha**: A first deployment of a service in a new subscription fails with MissingSubscriptionRegistration: the resource provider isn't registered. **Fix**: Register the provider namespaces the stack needs once per subscription (az provider register --namespace …), as part of the code that creates the subscription. - **Gotcha**: Australia East (New South Wales) has three availability zones; Australia Southeast (Victoria) has none. **Fix**: Run anything that needs zone redundancy in Australia East, and use Australia Southeast only as its paired region. ## Neon (serverless Postgres) [Neon (serverless Postgres)](https://cloudlesslabs.com/stack/neon/index.md): My default. - **Gotcha**: The pooled endpoint accepts 10,000 clients, but real concurrent transactions are about 90% of max_connections: roughly 94 at 0.25 CU, 377 at 1 CU. Queued clients time out after 120 s. **Fix**: Size compute for concurrent transactions, not client count. Use the HTTP driver for one-shot queries. - **Gotcha**: The HTTP driver has no interactive transactions. **Fix**: Use the WebSocket mode of the serverless driver when you need a session. - **Gotcha**: After 5 minutes idle, compute scales to zero and the next query waits a few hundred milliseconds. **Fix**: Fine for most apps. Disable scale-to-zero on production branches that need instant answers. - **Gotcha**: An urgent notification that writes to the database first fails when the database is slow, waking from scale to zero, or down. **Fix**: Send the alert first, then record it (or record it through a queue). The database keeps the history; it never gates the call. ## Next.js on Cloudflare Workers (OpenNext) [Next.js on Cloudflare Workers (OpenNext)](https://cloudlesslabs.com/stack/nextjs-on-workers/index.md): Use when it fits. - **Gotcha**: The build goes over the Worker size limit. **Fix**: Run `npx @opennextjs/cloudflare build` and read what's in the bundle; heavy server-only dependencies are the usual cause. - **Gotcha**: ISR pages serve stale content longer than expected when the cache sits in Workers KV, which is eventually consistent. **Fix**: Use R2 for the incremental cache, as OpenNext's docs recommend (and D1 for the tag cache). ## Node.js [Node.js](https://cloudlesslabs.com/stack/nodejs-javascript/index.md): My default. - **Gotcha**: The Lambda Node.js 24 runtime no longer supports callback-style handlers (event, context, callback). Old handlers break when you bump the runtime. **Fix**: Write every handler as an async function that returns its result, and test the bump on a staging alias first. - **Gotcha**: Lambda Node.js runtimes have end dates. Node.js 20 was deprecated on 30 April 2026; Node.js 22 follows on 30 April 2027, with updates blocked from 31 August 2027. **Fix**: Target nodejs24.x today (supported to April 2028), pin the runtime in Pulumi and plan one runtime bump a year. Node.js 26 is in preview on Lambda, due for general availability in November 2026. - **Gotcha**: Lambda ships an AWS SDK inside the runtime and updates it underneath you. AWS itself warns this can change behaviour. **Fix**: Bundle the @aws-sdk clients you use in the deployment package, as AWS recommends, so a runtime update can't change your SDK version. - **Gotcha**: Lambda treats a .js file as CommonJS unless told otherwise, and async work started at load time without await may finish during the first request instead of during initialisation. **Fix**: Set "type": "module" in package.json and use top-level await for start-up work (config, secrets, clients). AWS recommends ES modules for exactly this. - **Gotcha**: Cloudflare Workers are not full Node.js: child_process and worker_threads import but don't work, and several other Node modules are only partly supported. **Fix**: Keep code that needs them on Lambda. On Workers, use a compatibility date of 2026-08-04 or later, where Node.js compatibility is on by default. ## OpenAI Codex [OpenAI Codex](https://cloudlesslabs.com/stack/openai-codex/index.md): My default. - **Gotcha**: In the default workspace-write sandbox, network access is off. A test run against a staging URL, or a package install, fails until you allow it. **Fix**: Turn network_access on in the project's config for the projects that need it. Don't jump to danger-full-access, which removes the filesystem boundary too. - **Gotcha**: On Windows, computer use can't run in the background: it takes over the pointer and keyboard. It also can't approve OS security prompts or sign in as an administrator. **Fix**: Give it a dedicated machine or VM for long runs, and grant the app permissions (Screen Recording and Accessibility on macOS) before you start. - **Gotcha**: On Free, Plus and Pro, OpenAI may train on your content, Codex tasks included, unless 'Improve the model for everyone' is off. Business and Enterprise workspaces aren't used for training by default. **Fix**: Put client work on Business or Enterprise, or switch the setting off before the first task. - **Gotcha**: Web search and fetched pages can carry prompt injection, and the agent may follow it. **Fix**: Keep network access off unless a task needs it, and set web_search to disabled in sensitive repos. ## OpenCode [OpenCode](https://cloudlesslabs.com/stack/opencode/index.md): My default. - **Gotcha**: Out of the box, the agent may edit files and run shell commands without asking. **Fix**: Set edit and shell permissions to ask in the project's opencode.json, and commit it. - **Gotcha**: Its local server has had two remote-code-execution flaws (CVE-2026-22812, and an origin bypass fixed in 1.18.22): a malicious web page could drive it. **Fix**: Stay on the latest version, keep the server bound to 127.0.0.1, and set OPENCODE_SERVER_PASSWORD when you run serve or web. - **Gotcha**: Pay-per-token models plus an agent in a loop can burn money overnight. **Fix**: Give each OpenRouter key a credit limit that resets daily or weekly, one key per developer or experiment. - **Gotcha**: /share publishes the session at a public link, and some free models may train on your code. **Fix**: Commit "share": "disabled" for client repositories, and set OPENCODE_DISABLE_FREE_MODELS=1. - **Gotcha**: OpenCode's hosted Console was breached on 18 October 2026: OpenCode reported 1.4% of Console users with linked provider keys affected. The CLI was reported unaffected. **Fix**: If you linked provider keys in the Console, rotate them. Keep keys in your own environment rather than in a hosted console, each with a spending limit. ## OpenRouter [OpenRouter](https://cloudlesslabs.com/stack/openrouter/index.md): My default. - **Gotcha**: Calling it from the browser exposes your API key. **Fix**: Always go through a server-side proxy. A Cloudflare Worker is ideal: see the LLM proxy guide. - **Gotcha**: By default, requests are load-balanced across the providers serving a model, weighted towards the cheapest. Providers don't all serve the same quantization or support the same parameters, so the model you evaluated isn't always the one that answers. **Fix**: For anything you've tested, set provider preferences: require_parameters true, a quantizations filter, or an order of providers with allow_fallbacks set deliberately. - **Gotcha**: By default it may route to providers that collect or train on prompts. **Fix**: Turn off training providers in the account's privacy settings, and send provider.zdr true (zero data retention) or data_collection deny for client data. - **Gotcha**: Free model variants are limited to 50 requests a day (1,000 after buying $10 of credits) and 20 a minute. **Fix**: Never ship a :free model in a production feature. Use them to experiment. - **Gotcha**: OpenRouter falls back between providers, but not around itself: if OpenRouter is down, every feature behind it is down. **Fix**: Put Cloudflare AI Gateway in front, with a fallback that calls one provider directly with your own key. ## PagerDuty [PagerDuty](https://cloudlesslabs.com/stack/pagerduty/index.md): Use when it fits. - **Gotcha**: Without a dedup_key, every event opens its own incident: one fire, fifty calls. **Fix**: Set dedup_key to what makes the incident one (the site and the hazard, for example), and resolve with the same key. - **Gotcha**: Bursts of events can get HTTP 429 from the Events API. **Fix**: Send from your own alarm queue, so a throttled event is retried with backoff instead of lost. - **Gotcha**: SMS replies aren't supported for Australian numbers. **Fix**: Acknowledge by pressing a key on the call, or in the app. - **Gotcha**: Inheriting Opsgenie? Atlassian stopped selling it in June 2025 and shuts it down on 5 April 2027. **Fix**: Plan the move now, to PagerDuty or to Atlassian's replacement. ## PayPal [PayPal](https://cloudlesslabs.com/stack/paypal/index.md): Skip. - **Gotcha**: Tutorials and old answers point to IPN, Website Payments Standard and the v1 Payments API, which are all deprecated. **Fix**: Build only on the REST v2 APIs (Orders v2, Payments v2) with webhooks, and verify every webhook with PayPal's verify-webhook-signature call. - **Gotcha**: A payout isn't done when the API answers. The batch completes later, and an item sent to an email with no PayPal account sits unclaimed, then comes back to you. **Fix**: Give every batch a sender_batch_id (a retry with the same id is rejected as a duplicate for 30 days), process the item-level webhooks, and reconcile unclaimed items on a schedule. - **Gotcha**: PayPal can limit an account or hold part of your balance in reserve, and appeals are slow. A frozen balance stops your payouts too. **Fix**: Withdraw the balance regularly, keep business and identity details current before large batches, and keep a second processor ready to switch to. - **Gotcha**: Automatic currency conversion costs 3–4% above the base rate. **Fix**: Hold balances in the currencies you pay out in, and set the account to accept foreign currencies instead of converting them on arrival. ## pgvector [pgvector](https://cloudlesslabs.com/stack/pgvector/index.md): My default. - **Gotcha**: The HNSW index has to fit in memory, so large collections force a big, always-on database. **Fix**: Keep the hot data in pgvector and move the archive to S3 Vectors. Watch Neon's Lakebase Search, which keeps the index on disk. ## Playwright, Vitest, pytest [Playwright, Vitest, pytest](https://cloudlesslabs.com/stack/testing/index.md): My default. - **Gotcha**: Plain 'vitest' starts in watch mode in an interactive terminal and never exits. An agent that runs it waits forever. **Fix**: Make the test script 'vitest run', the single-run mode. Vitest only switches to run mode on its own in CI or a non-interactive terminal. - **Gotcha**: Each Playwright version needs its own browser builds. After an upgrade, tests fail until the browsers are reinstalled, and a fresh CI machine has none. **Fix**: Run 'npx playwright install --with-deps chromium' after every upgrade and in CI, and cache the browser folder (~/.cache/ms-playwright on Linux) between runs. - **Gotcha**: Parallel test workers eat a small machine. Playwright, Vitest and pytest-xdist all scale with the CPU, and on a shared box or CI runner the tests start timing out. **Fix**: Set worker counts explicitly: Playwright's docs use workers: process.env.CI ? 2 : undefined, Vitest has maxWorkers, pytest-xdist takes -n. Never run two heavy suites at once. - **Gotcha**: Playwright's test agents (planner, generator, healer) are generated files. After a Playwright upgrade they go stale. **Fix**: Regenerate them after each upgrade: npx playwright init-agents --loop=claude (or codex). ## PostHog [PostHog](https://cloudlesslabs.com/stack/posthog/index.md): My default. - **Gotcha**: Ad blockers block PostHog's own domains. PostHog puts the loss at 10 to 30% of events, depending on your audience. **Fix**: Send events through your own subdomain before going live: PostHog's free managed proxy, or a small Cloudflare Worker. - **Gotcha**: Identified events (tied to a person profile) can cost up to 4x more than anonymous ones, and any identify(), alias(), group() or setPersonProperties() call creates a profile. **Fix**: Keep person_profiles at 'identified_only' (the default), call identify() only after sign-in, and send backend events with $process_person_profile set to false unless they belong to a user. - **Gotcha**: A billing limit doesn't pause ingestion until you pay: everything over the limit is dropped and lost for good. **Fix**: Set limits as runaway protection only, watch the 80% email, and raise the limit before month end on anything you report on. - **Gotcha**: The region (US or EU) is chosen at sign-up. They're separate instances, and moving data between them later is slow. **Fix**: Decide at sign-up. For Australian clients with personal data, pick the region their privacy policy names, or self-host. ## Postmark [Postmark](https://cloudlesslabs.com/stack/postmark/index.md): Skip. - **Gotcha**: The unscoped npm package postmark-mcp was malware that BCC'd every email to an attacker. **Fix**: If you use Postmark with agents, the official package is @activecampaign/postmark-mcp. - **Gotcha**: Webhooks aren't signed (no HMAC). Anyone who finds the URL can post fake bounces or deliveries. **Fix**: Put basic-auth credentials in the webhook URL, allowlist Postmark's IP ranges, and check every MessageID against your own records before acting. - **Gotcha**: Webhooks are delivered at least once, retried for about an hour, and a 4xx other than 408 or 429 drops the event for good. **Fix**: Store the MessageID and the X-PM-Webhook-Trace-Id and skip repeats. Return 5xx, not 4xx, when your handler fails and you want the retry. - **Gotcha**: No idempotency on the send API: a request that timed out may still have sent. **Fix**: Record a send ID in your database before calling the API, and check the message log before retrying. ## Pulumi [Pulumi](https://cloudlesslabs.com/stack/pulumi/index.md): My default. - **Gotcha**: A vendor has no Pulumi package. **Fix**: Run `pulumi package add terraform-provider /`: Pulumi generates an SDK from any Terraform or OpenTofu provider. - **Gotcha**: Community providers are often maintained by one person (Neon, Resend, Clerk). **Fix**: Pin provider versions and upgrade deliberately. - **Gotcha**: New AWS services can land in @pulumi/aws-native before @pulumi/aws (Lambda Web Functions today). **Fix**: Use aws-native for those resources alongside the classic provider. - **Gotcha**: Pulumi doesn't bundle Cloudflare Worker code. **Fix**: Build with Wrangler, deploy the output with Pulumi, and give each Worker one owner. ## Resend [Resend](https://cloudlesslabs.com/stack/resend/index.md): My default. - **Gotcha**: Young company (founded 2023). **Fix**: Keep sending behind one small module of your own so a switch would be a day's work, not a rewrite. - **Gotcha**: The Terraform provider is community-maintained. **Fix**: Pin its version and keep templates in code, not in IaC. ## Route 53 and Cloudflare DNS [Route 53 and Cloudflare DNS](https://cloudlesslabs.com/stack/dns/index.md): My default. - **Gotcha**: Route 53 doesn't allow a CNAME at the zone apex. example.com can't be a CNAME to a CloudFront domain. **Fix**: Use an alias record (type A or AAAA) pointing at the CloudFront distribution or load balancer. On Cloudflare, CNAME flattening does the same job automatically. - **Gotcha**: Moving DNS providers with DNSSEC on breaks the domain: resolvers that validate DNSSEC return SERVFAIL once the old signatures and the DS record at the registry stop matching. **Fix**: Remove the DS record at the registrar first, wait (AWS suggests up to three days), switch nameservers, then re-enable DNSSEC on the new provider. - **Gotcha**: Only A, AAAA and CNAME records can be proxied by Cloudflare, and the proxy only speaks HTTP and HTTPS. A proxied record in front of SSH, a mail server or a database simply doesn't connect. **Fix**: Turn the proxy off (DNS only) for anything that isn't a website or API. MX and TXT records are always DNS only. - **Gotcha**: Proxied Cloudflare records have a fixed 300-second TTL, and Route 53 alias records to AWS resources use the resource's TTL. You can't set either. **Fix**: Plan cut-overs around five minutes of caching, and lower the TTL on non-proxied records a day before a migration. - **Gotcha**: Route 53 bills a hosted zone for the full month as soon as it exists, unless you delete it within 12 hours. Per-preview or per-branch zones add up. **Fix**: Use one zone per domain and records per environment, not a zone per environment. ## S3 + Glue + Athena [S3 + Glue + Athena](https://cloudlesslabs.com/stack/aws-athena-data-lake/index.md): Use when it fits. - **Gotcha**: Data partitioned by date, queried by customer: every query scans every partition. **Fix**: Keep the date partitions and sort each file by the second key (DuckDB COPY ... ORDER BY). Athena and DuckDB skip row groups by their min/max statistics, and Athena bills only the bytes it reads. For a real fix, use Iceberg tables in S3 Tables, partitioned by day(ts) plus bucket(N, customer_id), with sort or z-order compaction. - **Gotcha**: Partition projection and Glue partition indexes look like secondary indexes. They only speed up query planning. **Fix**: Use sorting, bucketing or Iceberg to prune data; projection to skip catalog lookups. - **Gotcha**: CTAS and INSERT INTO write at most 100 partitions per query. **Fix**: Write Parquet with DuckDB COPY (no limit, full control of sort order and file size), or split the job. ## Savings Plans and Reserved Instances [Savings Plans and Reserved Instances](https://cloudlesslabs.com/stack/aws-savings-plans/index.md): My default. - **Gotcha**: A commitment can't be cancelled. You can return a Savings Plan only within 7 days, in the same calendar month, and only if it's $100 an hour or less. **Fix**: Buy for the floor, not the average: the lowest hourly spend of recent months. Add more in small layers every quarter, so the plans don't all expire on the same day. - **Gotcha**: A Database Savings Plan and a Reserved Instance can't both discount the same workload, and Database Savings Plans are 1-year only, up to 35%. **Fix**: Reserved Instances for the big, steady instance that won't change; a Database Savings Plan for Aurora Serverless and anything likely to move. - **Gotcha**: Reserved Instances cover instance hours only. Aurora Serverless capacity isn't an instance, and storage, I/O and backups aren't covered at all. **Fix**: Check what the line actually is in Cost Explorer (usage type) before buying, and cover Serverless with a Database Savings Plan. - **Gotcha**: By default the discount floats to any account in the organization, so the plan you bought for prod can end up discounting someone's test cluster. **Fix**: Restrict the plan to the buying account, or group accounts with Cost Categories, when teams pay their own bills. - **Gotcha**: Destroying an aws.savingsplans.SavingsPlan resource only removes it from state. The plan keeps billing until its term ends. **Fix**: Buy commitments deliberately, by hand, after reading the recommendation. Don't let a pulumi up or a terraform apply spend money for years. ## Self-hosted MCP servers [Self-hosted MCP servers](https://cloudlesslabs.com/stack/mcp-servers/index.md): Use when it fits. - **Gotcha**: Unofficial MCP packages can be malicious (the fake postmark-mcp BCC'd every email to an attacker). **Fix**: Use vendors' official servers, pin versions, and read the code of anything community-made. - **Gotcha**: Token passthrough: a server that forwards the agent's token to the API behind it, or accepts tokens issued for something else, breaks the OAuth boundary. The spec forbids it. **Fix**: Accept only tokens issued for your server (check the audience), and call the API behind it with the server's own narrowly scoped credentials. - **Gotcha**: Since the 2026-07-28 revision, MCP has no sessions: every request stands alone. Servers that kept state per session, including Cloudflare's McpAgent, are on the legacy path, and a state handle (a cart or job ID) passed back as an argument is not proof of who's calling. **Fix**: Build new servers stateless (createMcpHandler on Workers). Store state under the user ID taken from the verified token plus a random handle, and reject a handle presented by anyone else. - **Gotcha**: A tool's output goes straight into the model's context, and the spec treats tool descriptions from untrusted servers as untrusted too. An email, ticket or web page a tool returns can carry instructions. **Fix**: Read-only tools first, writes as separate tools with their own scope, and the risky ones behind human approval. Request the smallest scopes and step up only when a tool needs more. ## Serper [Serper](https://cloudlesslabs.com/stack/serper/index.md): My default. - **Gotcha**: Prepaid credits expire 6 months after purchase. **Fix**: Buy the smallest pack that covers six months of real use, not the cheapest rate per 1,000. - **Gotcha**: Asking for 100 results still returns 10: in my test on 26 October 2026, num set to 100 gave 10 organic results for one credit. **Fix**: Page through with the page parameter, one credit per page, or use DataForSEO when you need the top 100. - **Gotcha**: An agent in a loop can search hundreds of times in one run. **Fix**: Put the calls behind your own small script that caps searches per run and logs every query, as the weekly report does. ## Slack [Slack](https://cloudlesslabs.com/stack/slack/index.md): Never. - **Gotcha**: On the free plan, only the last 90 days are visible, and data older than a year is deleted for good. **Fix**: Never keep a decision, a credential or a spec in chat. Write it down in the repo the same day, and pay for a plan if the history matters. - **Gotcha**: A Slack Connect channel shared with another company needs every organisation in it on a paid plan. **Fix**: Don't buy a workspace to talk to one client. Join theirs, or use email. - **Gotcha**: Since 29 May 2025, new commercially distributed apps outside the Slack Marketplace can read channel history only once a minute, 15 messages at a time. Internal apps keep 50+ requests a minute. **Fix**: Build agent integrations as an internal app inside the client's own workspace, not as a third-party app you distribute. ## Stripe [Stripe](https://cloudlesslabs.com/stack/stripe/index.md): My default. - **Gotcha**: Stripe doesn't guarantee order (invoice.paid can arrive before customer.subscription.created), sometimes sends the same event twice, and retries failed deliveries for up to three days in live mode. A slow handler that times out gets the same event again, and a month-start renewal spike can swamp it. **Fix**: Return 2xx at once and queue the work. Record processed event IDs and skip repeats. On any event, fetch the current object from the API instead of trusting arrival order. - **Gotcha**: Webhook signature verification needs the exact raw body. A framework that parses the JSON first breaks it. **Fix**: Verify against the body exactly as received, before parsing. On a Lambda function URL, that's event.body as a string, decoded first only if isBase64Encoded is true. - **Gotcha**: With EventBridge delivery, the partner event source must be associated with an event bus within 7 days, or AWS deletes it and Stripe disables the destination. Stripe also can't resend events to EventBridge by hand. **Fix**: Create the partner event bus in the same Pulumi change as the destination, and add an EventBridge archive on that bus before production, so you can replay after a processing bug. - **Gotcha**: A network retry on a create call can charge twice or create two subscriptions. **Fix**: Send an Idempotency-Key (a UUID per business operation) on every POST. Stripe keeps keys for at least 24 hours and rejects a reused key with different parameters. - **Gotcha**: A full secret key (sk_live_) can do anything in the account, payouts included, if it leaks or an agent misuses it. **Fix**: Use restricted keys (rk_live_): one per service and one per agent, each with only the permissions it needs. Stripe recommends them for anything you give an AI agent. ## Stryker mutation testing [Stryker mutation testing](https://cloudlesslabs.com/stack/stryker/index.md): My default. - **Gotcha**: The default test runner is 'command': it runs your whole npm test for every single mutant, with no idea which tests cover which line. On a real codebase that takes hours. **Fix**: Install the runner plugin for your framework (@stryker-mutator/vitest-runner for Vitest) and keep coverageAnalysis on perTest, so each mutant only runs the tests that touch it. - **Gotcha**: Playwright isn't a supported runner, so end-to-end suites go through the command runner: every mutant reruns the whole browser suite. **Fix**: Point mutate at the few modules the e2e suite really guards, and run it on a schedule, not on every push. - **Gotcha**: Incremental mode only notices changes in mutated files and test files. A dependency upgrade, an environment variable or a snapshot change isn't detected, so old results can be wrong. **Fix**: Run with --incremental in CI for speed, and a full --force run on a schedule and after every dependency upgrade. - **Gotcha**: Some mutants can't be killed because the mutated code behaves exactly like the original (a < b becoming a <= b where they're never equal). They drag the score down forever. **Fix**: Disable them in place with a reason: // Stryker disable next-line EqualityOperator: equivalent mutant. The reason shows in the report. - **Gotcha**: By default Stryker takes every CPU core on a machine with four or fewer, and all but one above that. On a small VPS or a shared CI runner it starves everything else. **Fix**: Set concurrency explicitly (a number, or a percentage such as 50%), and set thresholds.break so a falling score fails the build. ## switch-profile [switch-profile](https://cloudlesslabs.com/stack/switch-profile/index.md): My default. - **Gotcha**: Switching rewrites [default] in ~/.aws/config, so it changes the account for every terminal and every running process (Terraform, IDE plugins) on the machine. **Fix**: Use the sp shortcut, which sets AWS_PROFILE in the current terminal only. In scripts and agents, prefix every command with the profile (AWS_PROFILE=acme-prod aws …) and never rely on [default]. - **Gotcha**: Fuzzy matching is ambiguous in scripts: sp prod matches every client's prod. **Fix**: Use `switch-profile use ` in scripts. Ambiguous input exits with code 3 instead of guessing. - **Gotcha**: "Invalid start url provided" or invalid_grant at login usually means the portal's SSO region is wrong, not the URL. **Fix**: Run sp portals: it detects the right region and fixes it. - **Gotcha**: Accounts you reach through the console's account access manager don't work with aws sso login. **Fix**: Add them as console sign-in profiles (aws login), which needs AWS CLI 2.32 or later. - **Gotcha**: The device code doesn't reach the clipboard inside tmux. **Fix**: Set set -g set-clipboard on in tmux, or press q and scan the QR code. ## Tailscale [Tailscale](https://cloudlesslabs.com/stack/tailscale/index.md): My default. - **Gotcha**: A device's key expires after 180 days by default, and its connections stop. A subnet router that expires cuts everyone off from the VPC. **Fix**: Authenticate servers and subnet routers with a tagged auth key: tagged devices have key expiry disabled by default. Make the enrolment keys single-use or ephemeral and rotate them on a schedule, since a captured reusable key lets an attacker enrol devices. Declare the tags and keys in Pulumi. - **Gotcha**: Tailworm (25 October 2026) reported eleven unpatched findings: from one compromised device in the tailnet, Tailscale SSH and tailscale serve can lead to root on the servers. Fixes are in progress, with no release yet. Subnet routers aren't in any finding. **Fix**: Turn Tailscale SSH off where SSM or OpenSSH can do the job, tighten the policy file where you keep it, and keep every node updated. - **Gotcha**: Advertised subnet routes do nothing until approved, and Linux clients don't use them until told to. **Fix**: Add autoApprovers for the router's tag in the policy file, and run tailscale set --accept-routes on Linux clients. - **Gotcha**: Private hosted zone names and private AWS endpoint names don't resolve on your laptop: only the VPC's own resolver knows them. **Fix**: Add a split-DNS nameserver in Tailscale for those domains, pointing at the VPC resolver (the VPC's primary CIDR base + 2, e.g. 10.0.0.2), and allow port 53 to it in the policy. - **Gotcha**: One subnet router in one Availability Zone is a single point of failure for all private access. **Fix**: Run a second router in another AZ advertising the same routes; Tailscale fails over between them. Don't set --accept-routes on the routers themselves. - **Gotcha**: The free Personal plan is for non-commercial use. **Fix**: Put client work on a Standard plan in the client's own tailnet, not your personal one. ## Telegram Bot API [Telegram Bot API](https://cloudlesslabs.com/stack/telegram/index.md): My default. - **Gotcha**: Sending too fast returns HTTP 429: about one message a second per chat, 20 a minute per group, 30 a second in bulk. **Fix**: Send through a rate-limited queue and wait the retry_after the error gives you. - **Gotcha**: Bots can download files up to 20 MB and upload up to 50 MB. **Fix**: Run Telegram's local Bot API server: no download limit, uploads up to 2,000 MB. - **Gotcha**: A bot receives updates by webhook or by polling, never both, and the webhook only works on ports 443, 80, 88 or 8443 over TLS. **Fix**: Use a webhook on 443 in production (a Lambda function URL or a Worker is enough), and drop the webhook before polling in development. - **Gotcha**: "Task" and "earn" bots are a known scam pattern on Telegram, and its terms let it ban bots too. **Fix**: Link to your own website and privacy policy, never message people who didn't start the bot, and keep the brand consistent. ## Terraform / OpenTofu [Terraform / OpenTofu](https://cloudlesslabs.com/stack/terraform/index.md): Use when it fits. - **Gotcha**: State files hold every secret in plaintext, including values marked sensitive. Marking them only hides them from the CLI output. **Fix**: An encrypted S3 bucket with tight access for state. Keep secrets out of state with ephemeral values and write-only arguments, or use OpenTofu's state encryption with a KMS key. - **Gotcha**: DynamoDB state locking is deprecated in the S3 backend, and the S3 lockfile that replaces it is off by default. **Fix**: Set use_lockfile = true on the S3 backend and drop the DynamoDB table once every runner is on a version that supports it. - **Gotcha**: The dependency lock file pins provider versions only. Modules with a loose version constraint upgrade themselves on the next init. **Fix**: Commit .terraform.lock.hcl, and give every module an exact version. - **Gotcha**: Terraform has been under the Business Source License since August 2023 and part of IBM since February 2025. OpenTofu is the MPL-licensed fork, a CNCF project since April 2025. The two now release separately, with different features. **Fix**: Pick one per codebase, pin its version in CI and on every laptop, and don't mix the two binaries on the same state. ## Trello [Trello](https://cloudlesslabs.com/stack/trello/index.md): Use when it fits. - **Gotcha**: A free workspace allows 10 collaborators, counted across the whole workspace, guests and pending invites included. Go over and every board turns view-only, and automations stop running. **Fix**: Remove people from finished engagements, give each client its own workspace, or move that workspace to Standard. - **Gotcha**: Free workspaces also stop at 10 boards and 250 automation runs a month. **Fix**: One board per engagement, with lists for its phases. Keep automation to a rule or two. - **Gotcha**: The REST API allows 100 requests per 10 seconds per token and 300 per API key. Past 200 rejections in a window, every request on that key gets a 429 until the window ends. **Fix**: Sync scripts and agents batch their changes and back off on the first 429. - **Gotcha**: The MCP server connects to one workspace per connection, archives cards but never deletes them, and doesn't handle comments or attachments yet. **Fix**: One connection per client workspace. Use the REST API for comments and attachments in the meantime. ## Twilio [Twilio](https://cloudlesslabs.com/stack/twilio/index.md): My default. - **Gotcha**: Since 1 July 2026, a branded sender name (an alphanumeric sender ID) to Australian phones must be on the ACMA register, or the message shows as "Unverified" next to possible scams. **Fix**: Send alerts from an Australian mobile number: numeric senders aren't affected, and recipients can reply. Register a name only if the client wants one. - **Gotcha**: An Australian number needs a regulatory bundle (business and address details), reviewed in up to three business days, sometimes longer. **Fix**: Buy the numbers in the first week of the project, not the week before go-live. - **Gotcha**: A trial account only calls and texts verified numbers, and expires after 30 days. **Fix**: Upgrade before testing real escalation chains. - **Gotcha**: A call that reaches voicemail reports as answered. **Fix**: Count only a keypress as an acknowledgement, and turn on answering-machine detection to move on to the next person sooner. - **Gotcha**: Anyone who finds your webhook URL can post fake call results to it. **Fix**: Validate the X-Twilio-Signature header on every webhook. ## TypeScript [TypeScript](https://cloudlesslabs.com/stack/typescript/index.md): My default. - **Gotcha**: Node.js now runs .ts files directly, but only by stripping the types. Nothing is type-checked, and enums, namespaces with code, parameter properties, decorators and tsconfig paths fail at runtime. **Fix**: If you run TypeScript on Node.js without a build, turn on erasableSyntaxOnly in tsconfig so the editor rejects those constructs, and run tsc --noEmit in CI for the actual type check. - **Gotcha**: Under type stripping, a type imported without the 'type' keyword is treated as a real import and crashes at runtime. **Fix**: Turn on verbatimModuleSyntax and write import type for types. - **Gotcha**: TypeScript 7 ships without a programmatic API (Microsoft expects a new one in 7.1), so tools built on the compiler API, such as typescript-eslint, can't use it yet. **Fix**: Run those tools against TypeScript 6.0 installed side by side, as Microsoft suggests, until 7.1 lands. ## WhatsApp Business Platform [WhatsApp Business Platform](https://cloudlesslabs.com/stack/whatsapp/index.md): Use when it fits. - **Gotcha**: A new business starts at 250 people a day. Launch to a big list and you hit the wall on day one. **Fix**: Verify the business before launch, then ramp up: limits rise to 2,000, 10,000, 100,000 and unlimited as you use half of them with good quality. - **Gotcha**: You can open a conversation, or write after 24 hours of silence, only with an approved template. **Fix**: Design the templates first and get them approved before you build the flow around them. - **Gotcha**: An account can be locked or disabled for policy reasons, and a partner like WATI can't appeal for you. **Fix**: Subscribe to the account_update webhook, keep a fallback channel ready (SMS, email or Telegram), and appeal yourself in Business Support Home. - **Gotcha**: Since 1 October 2026, a bot that only replies is no longer free beyond 1,000 messages a month per number, and delivery stops without a payment method on file. **Fix**: Budget the replies, and put a card on the account before launch. ## Related - [Every tool](https://cloudlesslabs.com/stack/tools/index.md) - [Every use case](https://cloudlesslabs.com/stack/use-cases/index.md) - [Map of every Markdown file on cloudlesslabs.com, with what each one answers](https://ask.cloudlesslabs.com)