AWS Glue Streaming - streamName Required

Expected behavior

to be able to connect to a cross account kinesis data stream using the streamARN as on the cloud.

Actual behavior

streamName Required.

Additional Information

When I run the glue job on the cloud with streamARN and NOT the streamName it is successfully able to pull data from the stream. when I run the same code from the local docker aws glue streaming image it errors with the message:

pyspark.sql.utils.IllegalArgumentException: Stream name is a required field

Any assistance would be great.

It looks like you’re encountering an issue with AWS Glue Streaming in a local Docker environment where it requires a streamName despite using a streamARN successfully in the cloud. This seems to be a discrepancy between the local Docker AWS Glue image and the cloud environment.

Yea, If AWS provides the glue streaming image my would not base it of the current cloud 4.0 streaming image. I’m trying to do local development and connect to a cross account kinesis data stream. I took the code that works on the cloud but when I try and run it locally using the streaming image AWS provides it fails. Weird thing is that this image is 9 Months old and I cannot believe no one has run into this before. AWS Please get this fixed.