Since AWS launched its first service in 2004, it has innovatively led the way in the cloud computing space. For many years, FME users have been able to connect to key AWS database and storage services, as well as deploy their FME Server environment in AWS via FME Cloud. With the addition of a new dedicated team at Safe Software working on cloud desktop components, over the past year, we have focused on improving many of the existing transformers and adding support for several new services.

Below is an overview of the Amazon Web Services supported in FME 2020. This is part three of a three-part series on Cloud Services you can connect to using FME. Check out our other blogs about Google Cloud and Microsoft Azure.

Connect to Amazon Storage Services

 


Amazon S3
allows data to be accessed via an API.

FME’s S3Connector transformer allows you to access Amazon S3 buckets to upload, download, delete files and folders, or list file/folder information. As of FME 2019.2, this transformer replaces the S3ObjectLister, S3Downloader, S3Uploader & S3Deleter in a single transformer and adds support for any S3-compatible API via the new alternate endpoint and SSL verification parameters. Typical uses include:

 

Connect to AWS Relational Database Services (RDS)

Amazon’s fully managed database service includes SQL Server, PostgreSQL and MySQL/MariaDB.

 


Using FME’s readers/writers (
SQL Server, PostgreSQL, MySQL/MariaDB), you can read and write data to all of these services. These readers/writers are spatially enabled, which means you can read and write geometric data too.

If you need to execute SQL queries against one of these databases in the middle of a translation, you can use the SQLExecutor transformer instead of a reader/writer. This transformer can extract database records, execute database joins, perform SQL database table operations (create, drop, modify, truncate), and create/drop indexes or constraints before or after data loading.

DynamoDB

Amazon’s fully managed NoSQL database, Amazon DynamoDB, supports automatic scaling and geo-replication.

 


FME’s
Amazon DynamoDB reader/writer allows reading and writing large volumes of unstructured data to Amazon DynamoDB with a predictable low latency. This is typically used for storing JSON items in a highly available NoSQL DB, as well as for efficiently querying schema-free data via SQL with predictable performance.

Aurora

Amazon’s MySQL and PostgreSQL-compatible relational database for the cloud, Amazon Aurora, supports geo-replication and sub-second data access in any region. Additionally Amazon Aurora offers a serverless option which enables automica scaling.

 


FME’s
Amazon Aurora reader/writer allows reading and writing large volumes of data to Amazon Aurora with a very low latency on a global scale.

 

Connect to Amazon Notification Services

SQS

FME’s SQSConnector, which we recently reimplemented and improved, allows you to access the Amazon SQS service to send, receive, count and delete messages in an Amazon SQS queue. It replaces the SQSSender, SQSReceiver, SQSMessageCounter & SQSDeleter with a single transformer in FME 2020.0 and adds support for FIFO queues and advanced authentication options. Workspaces using the deprecated transformers can be easily upgraded to use the new functionality. Typical uses include:

FME also supports Amazon SQS as a Trigger and an External Action in FME Server Automations, enabling you to integrate your AWS environment with complex automated workflows.

 


The
Amazon SQS Event Received trigger allows you to connect to an Amazon SQS Queue to receive and delete notifications with FME Server and trigger Automations.

The Amazon SQS External Action allows you to add messages to an Amazon SQS queue from within an Automation.

Typical uses include:

SNS

FME’s SNSSender allows you to send messages to an Amazon SNS topic. FME also supports Amazon SNS as a Trigger and an External Action in FME Server Automations, enabling you to integrate your AWS environment with complex automated workflows.

 


The
Amazon SNS Event Received trigger allows you to subscribe to an Amazon SNS topic to receive event notifications in FME Server and trigger Automations.

The Amazon SNS External Action allows you to send a message to an Amazon SNS topic from within an Automation.

Typical uses include:

 

Connect to Amazon Data Analytics services

Amazon’s fully managed data warehouse, Amazon Redshift, supports automatic scaling and geo-replication and has extensive support for geospatial data.

 


FME’s
Amazon Redshift reader/writer allows reading and writing large volumes of data to Amazon Redshift. As with most of our database/data warehouse formats, you can also use the SQLExecutor transformer instead of a reader/writer if you need to execute SQL queries against Amazon Redshift in the middle of a translation. We’re excited to share that we’re working on adding full support for the new geometry data type added to Amazon Redshift (announced in late 2019) to our future FME release this year. Stay tuned to safe.com as more information and content becomes available.  

Amazon Athena, Amazon’s serverless interactive query service, allows users to analyze data in Amazon S3 using standard SQL. Additionally, Amazon Athena supports a long list of spatial functions that can be queried by FME.

 


FME’s
AthenaConnector, new in 2020, can be used to send queries to Amazon Athena, store and reuse saved queries and store results of queries. This is typically used for:

Amazon’s Managed Streaming for Kafka (MSK) is a fully managed, real-time data ingestion service with Kafka compatibility.

 


FME’s
KafkaConnector can be used to send and receive data from Amazon MSK. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read. Typical uses include:

 

Connect to Amazon AI Services

Amazon’s pre-trained Rekognition model allows you to quickly classify images into millions of predefined categories.

 


FME’s
RekognitionConnector transformer, new in 2020, allows you to interact with the Rekognition service. You can use this to detect and classify multiple objects in an image, detect faces in images, and read printed and handwritten text.

Amazon’s pre-trained Comprehend model allows you to analyze text for the language type and the sentiment.

 


FME’s
ComprehendConnector, new in 2020, allows you to interact with the Text Analytics API. You can use this to understand the overall opinion, feeling, or attitude sentiment expressed in a block of text, as well as to detect the language of the text.

 

Connect to Amazon Internet of Things

AWS IoT Core allows you to securely connect, manage and ingest data from millions of IoT devices via MQTT.

 


FME’s
MQTTConnector transformer, new in 2020, can be used to send and receive data from AWS IoT Core. This transformer supports both stream and batch mode. If you initiate in stream mode, a persistent connection is created and hundreds of messages per second can be read. Typical uses include:


Our customers have been using FME to connect to Amazon Web Services for years. Because FME Cloud is hosted on AWS, these services are logical choices for technology professionals looking to take advantage of data gravity. With the recent improvements made by our cloud development team, we hope to continue to remove barriers between your data systems and applications and facilitate your cloud-based workflows.

As always, let us know what you think about these transformers and connectors by sharing your thoughts in the FME Knowledge Center.

About Data About FME AWS Cloud Services FME Cloud

Gerhard Fischl

Gerhard is a FME Cloud Technology Expert at Safe Software. He is one of the top soccer players during lunch time soccer and is an avid bicyclist.

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Related Posts