Faced with migrating 18 years worth of cases from Bugzilla to Jira, the Safe Software development team built four FME workspaces to do the job. Read on to learn how FME tackled this massive data migration undertaking.
Times are changing, and so are we. Over the past year and a half, the Safe Software development team has been working hard to migrate all the support cases that were logged in Bugzilla for the past 18 years into Jira. 18 years! That’s the same amount of time it’s taken me to get from learning to read in grade 1 to writing this blog post for Safe Software.
We’re opting for Jira as our primary project management platform for a few reasons. For one, it’s not just for developers. Jira can be used by a variety of teams for planning their projects. Within Jira, you can also integrate other cloud applications, which is very convenient! For us, being able to connect Salesforce to Jira is a huge plus. Overall, it’s very supportive of the agile framework as it allows for adaptive planning and collaborative efforts.
Why migrate 18 years worth of cases, you may ask? We want to ensure we keep records of all the changes, cases, and bugs we’ve logged. Some are still open, many are closed, but overall, having all this information ready and in the same platform will help our developers identify how to move forward with new tasks.
Now, before I jump into how this all took place, let’s start off by explaining a few of the differences between Bugzilla and Jira.
What is Bugzilla? What is Jira?
Bugzilla is a free, open-source, web-based program used by development teams for tracking bugs and testing tools. Thousands of companies use it today. Safe Software has been using the system since the beginning. It’s been extremely helpful with organizing and managing a variety of issues across the board, whether they be internal or external. Simple, straightforward, and easy to use. The perfect combination for any product!
Jira, developed by Atlassian, started as a software development tool like Bugzilla. However, it has long since grown into something more. Many teams, not just developers, can use Jira to track the stages of their projects easily. This is perfect for companies that are trying to move towards a more “agile” environment (like us!). Teams can customize how they want their Jira environment to work so it is best suited for them. Team members can report issues, assign tasks to themselves or others, and participate in discussions. Having all teams within a company using the same system for bug tracking, development, and project management means that everyone will be very familiar with the same tool and can communicate with other teams all in the same space.
Why FME?
Jira has two licensing options: server and cloud. There are pros and cons for each of these options, and after weighing them out, the Safe Software team decided on Jira’s cloud license. However, Jira’s built-in Bugzilla-to-Jira migration tools are only compatible with the server platform and are not available for cloud. If only we could automate the movement of data in some other way… oh right, that’s our expertise!
Naturally, we looked into how we could “eat our own dog food” and use FME as our main tool for the process of migrating issues to Jira. While writing a script was also an option, it was ultimately not the best route for the development team. Since the process was going to be complicated no matter what, the team opted for FME because of its visual interface. Being able to see how conversions and transformations were taking place made the workflow easy to understand and to edit when changes were needed. Rather than needing to remove code and up-keep syntax (which we all know can be a major pain), in FME you can add or delete a transformer by dragging and dropping the icon to where you need it in the workflow and then you can set the parameters within.
FME not only saves time by running workflows with large amounts of data efficiently, but it also gives you a lot of flexibility as you can customize transformations for your exact needs.
Let the dogfooding begin!
Building a Workflow
Iain McCarthy and his team (including some amazing co-op students!) took on this project. Together they were able to create four workspaces, each with their own specific function, that moved information from Bugzilla to Jira.
Before creating the workflows, a few main challenges needed to be considered:
- Jira’s Cloud Web Import Wizard only connects to CSV, JSON, and Trello
- There were about 85,000 issues to move, which equates to about 1 GB of data
- Data was coming from multiple Bugzilla tables that needed to be joined
FME was used to read Bugzilla data, which is stored using the MySQL database structure, and write it out in JSON. This data was then uploaded directly into Jira using the Web UI while their REST API was used to double check the work that was done with processing the data in FME. That’s the general A-to-B explanation of what was done, but it wasn’t really all that simple. There was a LOT of processing involved – string processing, list processing, you name it. The workspaces were also used to add updates to Bugzilla to help link issues between Bugzilla and Jira for future reference.
Can you guess how many transformers were used? 500. That’s right, FIVE HUNDRED. That’s insane.
Time to run these bad boys.
Understandably, Jira wasn’t able to import all 85,000 issues at once. To combat this, the workflow was run multiple times, with each run processing about 5,000 issues at a time. Processing in smaller groups gave the team more control over how various issues were being processed. In addition to the 500 transformers, there were also 25 parameters that needed to be set before running the workflow on each group.
Each group took about 20-30 minutes to be processed. Rather than running the workflows on a set schedule, each run of the workflow was done on demand so that parameters could be set appropriately for each group.
Finally, “40 hours of FME processing, one million SQL operations, a quarter-million REST calls and 750Mb of JSON later….” the migration is complete!
What Happens Next?
Well… now what? It feels a bit sad just to have this workflow saved on a computer as a memory. Instead, we’d like to share our creations with others. The plan now is to clean things up a bit, add in a few more comments, and upload parts of the workflow onto FME Hub. Many sections of the workflow are generic enough that they can be used within other data migration workflows, like moving Trello cards and lists to Jira.
We hope you enjoyed learning more about our “eating our own dog food” experience. This project was strenuous but very rewarding for everyone here at Safe Software. If you have any ideas or are inspired to tackle your own data migration projects with FME, share your thoughts with us in the comments below!
If you’d like to learn more about this process, be sure to sign up for the webinar!
Amanda Schrack
While Amanda's background is in environmental science and GIS, she now writes content for safe.com. Looks like writing all those research reports paid off! In her spare time, Amanda likes to make crafts, watch documentaries, and learn about bugs (the kind with six legs).
