Wait, hold up, before we dive in, you might be sitting there wondering,
“I’ve never heard of this FME Community, what is that?”
Well wonder no more, the FME Community is a place for everyone! The goal of this platform is for FME users and Safe staff to come together to share ideas and users’ stories, network, and play games. Most importantly, it’s also a space to get product support by starting discussions through asking (or answering) questions and browsing our extensive knowledge base.
Why did we move the FME Community from AnswerHub to Salesforce?
The FME Community has undergone many ‘glow ups’ over the years, and some of our long term FMEr’s may even recall the original fmepedia back in 2005! We’ve come a long way since then and in 2015, we underwent rejuvenation to launch over the hosting platform, AnswerHub, under a new name ‘The Knowledge Center’. Once again, it came time for a change, and in July 2020, we landed in a new Community backing onto Salesforce.
Salesforce is the Customer Relationship Management (CRM) Program used across many of the departments at Safe Software. For the Customer Success Team, this is how we already interact with our users through support cases, and recently how we communicate via our Live Chat support. The ability to consolidate all this customer information into a single source was a key reason for the change.
Why did we use FME to perform the migration?
I know what you’re thinking… Of course we chose FME as the star of the show – we’re Safe Software! And we don’t blame you. But putting the very real biases aside, there are several equally real reasons to use FME for this kind of project!
FME, the All-in-one Tool
Moving a community from one platform to another involves a ton of moving parts. From a data perspective, two key requirements were: preserving legacy information and accurately and efficiently loading that information into the new platform. To achieve this, we needed a tool that could:
- Interface with databases, from dev to staging to production environments
- Leverage REST APIs both programmatically and dynamically
- Provide extensive data transformation, QA/QC, and data validation capabilities
- Enable us to rapidly develop, version, and share our work
- Automate testing and production processes
- Accurately log and notify stakeholders of errors every step of the way
How we did it!
How did we manage to move, 1,642 articles, 3,045 ideas, and 18,450 questions over to a new platform? Well, in addition to some trial and error, there were three main stages of transferring our community to a different system:
- Fetching data from AnswerHub to store in a working database
- Pushing data from this database into Salesforce
- Automating with FME Server
Step 1: Fetching data from AnswerHub
This step was fairly straightforward. The AnswerHub REST API in combination with the HTTPCaller in FME, was integral in fetching all the data stored within Answerhub. As a team, we spent some time determining a standard schema to use when storing the data from AnswerHub in an Aurora database. Using a staging database like this helped us organize the data to be carried over to the new system.
Step 2: Pushing data to Salesforce
Now, this is where the fun really began! Once we had gathered all this information, it was time to learn how to push it to Salesforce. We used FME to do a lot of validation, such as catching HTML tags not supported in Salesforce, updating images, and mapping our existing topics to the new ones in the community.
Step 3: FME Server Automation
We used FME Server Automations to run our workspaces for the final migration. This helped ensure that our many workspaces would be triggered and run in the correct order automatically. We found the Merge Action within Automations particularly useful for this. Using FME Server also allowed us to offload the work from our normal machines to powerful AWS machines we had set up. If you are curious about Automations, check out our webinar Getting Started with Enterprise Automations.
Challenges and Top Tips
Looping in a Custom Transformer
Like most API requests, the AnswerHub API has a limit on the number of items returned in a single call. Therefore we had to find a method to iteratively work through each result page until every item was retrieved. Making use of looping within a custom transformer was the perfect solution! This functionality enabled us to make the same call over and over again simply changing the page number at each loop.
AnswerHub Rest API Pagination Custom Transformer
Adjacent Attribute Handling
One challenge we faced was replacing multiple different image or URL links within a single HTML body. The image below shows one such example that contains links to AnswerHub storage for different images.
Example of an HTML Body where multiple string replacements were required
The AttributeManager has an option to ‘Enable Adjacent Feature Attributes’ allowing us to easily replace multiple images or URLs in a single question, article or idea body.
Replacing multiple URLs in a single body of text using Adjacent Attribute Handling
Using Python
Due to the limited amount of HTML tags supported by Salesforce, a lot of them had to be stripped. To solve the problem of modifying unstructured data in place (eg. a large HTML body) and the variations that arose. We leveraged a Python library that’s shipped with FME and actually used in the HTMLExtractor: BeautifulSoup.
Eliminate various invalid tags
Natural Language Processing
In the past, users were able to tag topics at will on their material. In the new Community, this is restricted to a handful of clearly defined categories. One of the largest data conversion tasks we faced was mapping these topics (974!) to the 7 new capabilities. This was the perfect opportunity to experiment with FME Natural Language Processing (NLP) transformers. This approach saved us countless hours of manually labelling and matching all the existing topics to our new capabilities.
Example of a training dataset for the NLP Model
If you’d like to learn more about some of the processes we’ve mentioned, tune in to our upcoming webinar on September 17. Reserve your free spot today!
Laura Wu
Laura Wu has been with Safe Software since 2011 and is currently a Technical Experts Team Lead. She loves working with the amazing network of partners and clients helping them solve problems with FME. Away from work, Laura enjoys exploring the great outdoors and attempting to care for a growing collection of indoor plants.
Nathan Hildebrand
Nathan is an FME Server Technology Expert with a background in GIS, English, and forestry. Before Safe, Nathan spent time replenishing our forests with trees (he’s planted over 250,000!). When Nathan’s not in the forest, and not answering your questions, you might find him puttering through some Mozart on his French horn.
Sienna Emery
Sienna is one of Safe’s FME Server Technology Experts. When not in the office, Sienna can be found competing for the fastest time in spin class or playing board games with friends.
Holly Coxon
Holly is an FME Server Technology Support Team Lead at Safe Software. Among other things, she provides technical support, creates content for the Community, presents webinars and contributes to product development through functional testing and sharing customer feedback. Holly spends her weekends exploring the great backyard of British Columbia.
