With pre:Invent 2023, AWS launched a feature for which I had been eagerly anticipating and had also been advocating towards AWS for roughly 6 years: Amazon Kinesis Data Streams launches cross-account access with AWS Lambda.
Until this feature was launched, you could not easily trigger a Lambda from a Kinesis Stream in another account, something you would commonly encounter in architectures using Kinesis and Lambda as building blocks.
The recommended way in this scenario was, until then, to use a blueprint by AWS called aws-lambda-fanout. I also made use of this blueprint, and after some learnings with it, we rewrote the sample in Go and simplified it by supporting fewer targets.
After a year or two, we had not much hope that AWS would launch something to make this obsolete, so we started improving the ways on how we configure and roll out this aws-lambda-fanout. For example, we centralized the configuration and were able to generate diagrams illustrating how accounts interact with each other.
After another year, we started implementing basic event filtering for our fork of aws-lambda-fanout, which some of our services relied on.
After more time of learning, we again improved the way on how the solution was deployed into the AWS organization. We improved the resilience significantly by making it impossible for targets to get either deleted or denied by IAM while there was an active fanout deployed.
Over the whole time, we provided feedback to AWS and requested a way to overcome this solution because it caused maintenance effort, runtime cost, and felt like a pretty unique solution that should not be necessary.
And then, AWS announced the launch of Amazon Kinesis Data Streams launches cross-account access with AWS Lambda. It was really nice to see that AWS reacts to customer feedback. But it was also sad because we realized they are too late. They launched a solution for the issues we had 6 years ago. But we could, of course, not wait for AWS - we have a business to run. So we built solutions that diverged now too much from what AWS finally delivered and which solve additional problems as well. So we will probably not make use of the feature by AWS - even though we requested it, and this feels kind of frustrating for me - but it was just too late.
So my learning is, that it’s not enough to just solve your customer problems eventually - you need to do it fast, otherwise the problems of your customers will change - even if you might believe you control their ecosystem.