A Data Migration Is Never Just a Data Migration

Alex Meadows, a Data Architect with five migrations under his belt, on lift-and-shift versus rearchitecting, why data quality is still not a priority, and how organizational culture can make or break a migration.

With 18 years of experience in the data space, Alex Meadows has built a career navigating the complexities of data across a wide range of industries, from plastics and software to social media, agriculture, and med tech. His passion for data engineering, architecture, and modeling has made him a trusted expert in tackling high-stakes migration projects–and he’s completed five of them to date. 

"I’ve been around the block a time or two," Alex says with a modest confidence as we begin our conversation, which ran way overtime as we got caught up in the nuances of data quality, the intricacies of cloud migrations, and the often-overlooked human elements of these projects. His hands-on approach and strategic foresight have helped him lead teams through the challenges of transforming legacy systems into modern, scalable solutions.

Our interview explores three core themes: the pitfalls of lift-and-shift migrations, why data quality is a critical yet overlooked aspect of migrations, and how organizational culture can make or break a project. Along the way, Alex shares stories that illustrate his thoughtful approach to solving data migration challenges, highlighting how technology, politics, and people intersect in this complex process.

The lift-and-shift trap

We first connected with Alex on Data Bluesky, where a call for interviewees about data migrations caught his attention. Alex’s response immediately stood out:

The divide between the "lift-and-shift" and "rearchitect first" camps is a contentious topic in the world of data migrations, and Alex’s concise, pointed reply made it clear that he had both strong opinions and hard-won lessons to share. We knew we had to speak with him—and the conversation that followed didn’t disappoint.

Datafold: Circling back to how this all got started–your Bluesky "skeet"–it seems like you have very strong views on lift-and-shift versus rearchitecting, which I'm really excited to get into.

Alex: So just to be clear, it's not super strong. It really just depends on what you're trying to do. It might help if I give a little bit of a background. 

Alex recounted a particularly challenging case from about a decade ago when his team worked with a large agricultural company. The company wanted to move their legacy inventory system from Microsoft SQL Server on a traditional data center setup to Postgres on Amazon Web Services RDS. However, instead of fully rearchitecting the system as Alex’s team suggested, the client opted for a full lift-and-shift approach—moving the existing setup into AWS as-is.

Datafold: Tell me about why things went wrong with the lift-and-shift approach. 

Alex: Performance was awful because they were thinking in terms of a data center, not about how the cloud can boost performance as you need it and shrink back down. They didn’t have any scalability. It was just like the equivalent of bare metal.

Datafold: At what point were you brought in? Did you have any say in the initial decision?

Alex: So my group actually came in about two years into the project. They had been doing it for two years, and they had actually built it once, scrapped it, and we had to build it again. And it was still dictated-it had to be SQL Server; it had to be all of this stuff. We also had replication and had to support the legacy system. 

Datafold: Once your group entered the picture to essentially restart the process, what was the main barrier to rearchitecting this? Politics? Fear?

Alex: All the above. It was a lot of fear, lack of education, and understanding. Because you're talking about a group of people who were still thinking data centers were the main thing, and you had to–how familiar are you with data centers and how you would budget for them?

Datafold: Not at all. What's the mental model here? 

Alex: So the mental model for a data center is you have to plan for three to five years, because you get to spend a certain amount each year. You want to maximize that spend, because it's "use it or lose it." So you buy hardware and infrastructure above what you need to make sure you've got a comfortable cushion. It might be a lean year next year and you want to get whatever you can, while you can, so it doesn't blow up on you. So they're taking that mentality and trying to apply it to the cloud, which doesn’t work.

Datafold: In an ideal world, how would you have approached this migration?

Alex: I would have done the rearchitecture ahead of time, and started with the four elements. For instance, switch over to Postgres on AWS RDS. You can set up your cluster however you need to, but you can be smart about it from the get go. Having images of things is also super critical. Don't try to set up purely on an EC2 instance and just leave it there. There are pros and cons to both. However, they were using EC2 instances across the board because they were saying, ‘We need servers of X compute.’ They weren’t planning for that scale-up, scale-down kind of architecture.

Datafold: Right. How long was the project, the lift and shift part of it?

Alex: It took about a year. I mean, that went pretty well considering you were starting with absolutely no cloud. They had to set up AWS accounts, deal with all the VPC connections—the whole nine yards. They were literally greenfielding it. Then they had to test it out and prove that it would do what they needed it to, which we already knew, but they had to go through that for politics’ sake.

Finally, we got started, which involved an education shift because they had to hire more people to be the AWS DevOps team. Then came educating everybody on how to work with AWS. A lot of that was the bulk of the effort.

The overlooked priority–data quality

For Alex, data quality is the single most critical—and most overlooked—aspect of a successful data migration. While testing pipelines and applications for functionality is essential, Alex stresses that ensuring true parity between data sets must take center stage.

Datafold: Do you recall anything about the testing strategies used to determine parity? Because I understand that that’s like a very unsolved problem, basically.

Alex: Well, the real trick is you have to go a couple of different routes. So you’ve got, of course, your unit testing itself. You can make sure that everything’s working the way it should from a code standpoint—that’s a given. 

Where it gets tricky is you have to be able to do quality checks on your data set. And it’s not just making sure the pipelines work the way they’re supposed to, or that the app works the way they’re supposed to. It’s, ‘Do you have true parity between your data sets?’ Right? Because people are worried about their data.

Datafold: Right. So the tricky part is about ensuring true parity between data sets.

Alex: Yes, especially when you’re still making changes to the application while you’re going. But you have to have a good test suite. You have to have good test automation; otherwise, it’ll blow up really quick if you’re not careful.

This, Alex explains, is where most companies stumble. The challenge isn’t just building the right tools but cultivating the right mindset.

Datafold: I’ve heard this before—that you need a really good test suite. What does that look like? Because there’s no standardized protocol, right? Like it’s dependent on the engineering team and how good they are at foreseeing issues. 

Alex: Yeah. I was just having this conversation a couple of days ago, actually. It’s not just about the tooling; it’s also about the mindset. You have to have people that actually think quality. Because the argument I was having the other day was someone saying, ‘Oh, the devs can do their own testing.’ I’m like, yes, to a point.

You’ve got two sides of it: A dev can write tests for their code, totally. They can write unit tests, no problem. But another dev can come in, and they will see things that the first dev didn’t see. But they’ve also been in that code long enough that they’re going to have blind spots too. 

You have to have someone who is a QA, who thinks like a QA, because they will go in and break things—you want that. They are your friends. They will come in and break things. 

You know, the joke about the QA who goes into the bar and orders a beer, orders a Coke, 99 beers? 

Datafold: No, I don't know that one. 

Alex: Let me find it. 

Datafold: This is hilarious. 

Alex: So QAs act as the test for the bathroom, right? They think differently than a dev ever will, unless the dev has also done QA work or been through the fires with a QA. 

That’s the thing. They’re not just testing functionality—they’re testing assumptions. So it’s not just about the tooling. 

Datafold: That makes sense. 

Alex: Now, as to your question on what tools to use, I think it’s a multi-pronged approach. It's testing your app, testing your pipelines, and your back end, like your APIs and that kind of thing. You have to have that.

And this is the critical one that everyone misses in my mind, and that is data quality. They do not put data quality tooling as a priority. It’s hard enough to get most companies to do QA, right? They always treat QA as a second-class kind of thing, like, ‘Oh, we’ll get to it eventually.’ And they never do because they want to just keep developing stuff. But QA is super critical. And they never do because they want to just keep developing stuff. 

But QA is super critical for data quality, and I don't want to call it data governance, but data governance is definitely part of it too, right? And I will get on my soapbox on this one, but yes, data governance is an important thing too. 

Datafold: It seems to me like data quality is the target outcome of a data migration. Why would it not be a primary concern?

Alex: Because a data migration is never just a data migration. It’s usually tied to something else—an application, a process. Data by itself is never the big thing.

My passion is data platforms—building these data ecosystems so people can have all the data they ever wanted to answer any questions they might need over time. But most companies, almost all the companies I’ve worked for, will never want to build that. They’ll say they do, but they never want to. They don’t want to put the money or effort into actually doing proper data governance to find out what the data is and how you know the quality around it and all that. It’s always a fight.

Data migration problems are people problems

Data migrations may seem like a purely technical challenge, but as Alex points out, they are fundamentally human endeavors. The same patterns of miscommunication, resistance to change, and lack of strategic prioritization repeat across industries, creating avoidable chaos.

Datafold: It must be very satisfying to see them all come together in an organized fashion. 

Alex: If I can finally do it, yes. But the problem has always been politics and getting buy-in, because these types of projects–whether you're talking about a data lake house or a data platform–they're massive, right? They take years to do, and people don't want to hear that. It gets really nasty, really quickly, especially when they don't want to get the headcount to actually speed it up.

How familiar are you with data warehousing in general?

Datafold: Probably, like, at the 200-level.

Alex: Okay, so why don't people talk more about data warehousing? Why has it gone away a little bit? It's not just the cost. Are you familiar with TDWI? They used to be the Data Warehousing Institute. Now they're called Transforming Data With Insight. They are a vendor-agnostic education group that teaches all about data—data modeling, data quality, data warehousing, big data, you name it. I highly recommend diving in there.But what I want to say–and this is a while ago now–I think they reported that about 70% of data warehousing projects fail.

Datafold: I believe the same is true for data migration projects.

Alex: Yeah, and there’s a lot of reasons why, but the big one is time to delivery. It takes all this time to get where it needs to be. It takes all these resources and money because, while storage is cheap, compute is not. It costs a lot to process data. Like, right now we’re in GCP, and to start from nothing and process six terabytes of data, you’re talking thousands of dollars. And for a lot of businesses, that’s just not something they have an appetite for. 

Another factor is a lack of education. People have a tendency toward instant gratification—they don’t realize that if they spend a little bit here, it’s going to pay dividends down the road.

The same is true for data migrations or any big data project like this. People don’t understand it. They say, "Why is it taking so long?" Well, you said you wanted everything—you didn’t narrow it down, so we’re giving you everything. Is there a specific area you want to focus on? And it’s always, "I want it all, but I want it tomorrow." That’s always been my experience. 

Now, if you follow an agile approach, you can layer the cake, deliver good quality, and run with it. It works really, really well because you can tackle specific topic areas in a particular source or across sources. 

But there’s also the data quality problem, which we’ve already talked about quite a bit being a huge issue. People don’t want to put any resources into actually monitoring, measuring and handling data quality.

And then people skip the data platform or data lakehouse layer. When you go straight from the source to the end solution, you end up with spaghetti. There’s a lot of techniques to handle that correctly–you can be smart about it. But the people who are doing the work tend to not know those things, so they're rushing it. They’re not building frameworks the right way. They’re piecemealing it as they go, and the business logic is all over the place, data quality cleanup has to be repeated constantly. 

There’s all of these things that factor in that make these projects nasty—just so nasty—because you get people like me to come back in and try to clean it up, but by that point, we’re already under a crunch.

Datafold: It sounds like you've seen a lot.

Alex: You tend to. And here’s another factor I’ll throw out—this isn’t just industry-specific. Every single company I’ve worked with has had similar problems and issues. You could literally take away the company names, boil it down, and find the same patterns almost everywhere I’ve been.

Alex’s observations highlight a universal truth: technical challenges are rarely the true roadblocks in data migrations. Instead, it’s the interplay of human factors—short-term thinking, organizational silos, and a failure to invest in long-term quality—that creates failure patterns across industries. This underscores the need for leaders to prioritize not just the technical, but also the cultural and operational aspects of these projects.

On becoming an expert, context, and constraints

As we wrapped up our conversation, we wanted to know how Alex had developed such a systematic and thoughtful approach to data migrations. Was it the result of strong mentorship, formal training, or instincts honed on the job? His answer was candid and telling.

Datafold: I’ve got one more question to finish this off. So you clearly have a very systematic approach to migrations. Is this something you’ve learned through experience, or did you have a mentor or a specific protocol to follow?

Alex: Trial by fire. Because a lot of what I've done is self taught. I have worked really, really hard to be an expert in my space, and I take great pride in what I know. It disappoints me that more people don’t do that. If you’re in a space, take a little bit of time each week to learn new things. That’s something I try to do and encourage others to do, to varying success. 

Datafold: How do you find new things to learn?

Alex: Follow the best practices, find the folks who are the experts in those spaces, and combine their stuff to make a better thing. 

Every piece of that is a tool in the toolbox. There are very, very few things I will say, “Thou shalt do,” when it comes to building this kind of stuff. It really depends on the project—what’s the data you’re working with, how are you handling historization? 

And then, when you’re migrating this data, how do you migrate a piece at a time? Or do you try to do it all in one go? It really just depends on what you're trying to migrate, right? Because some things just won't tolerate it.