Expected behavior
to be able to connect to a cross account kinesis data stream using the streamARN as on the cloud.
Actual behavior
streamName Required.
Additional Information
When I run the glue job on the cloud with streamARN and NOT the streamName it is successfully able to pull data from the stream. when I run the same code from the local docker aws glue streaming image it errors with the message:
pyspark.sql.utils.IllegalArgumentException: Stream name is a required field
Any assistance would be great.
It looks like you’re encountering an issue with AWS Glue Streaming in a local Docker environment where it requires a streamName
despite using a streamARN
successfully in the cloud. This seems to be a discrepancy between the local Docker AWS Glue image and the cloud environment.
Yea, If AWS provides the glue streaming image my would not base it of the current cloud 4.0 streaming image. I’m trying to do local development and connect to a cross account kinesis data stream. I took the code that works on the cloud but when I try and run it locally using the streaming image AWS provides it fails. Weird thing is that this image is 9 Months old and I cannot believe no one has run into this before. AWS Please get this fixed.