There’s probably no greater technology that can benefit farmers more currently, than the tools and services that allow for data collection and analysis across a farming operation. Consider how calorie counting can help folks lose weight by simply gaining better visibility over what they are eating. It’s similar having data collection across all parts of a farming operation can reveal where resources such as labor and capital are being spent at a greater degree of accuracy. This allows for cost savings and greater yields as growers move closer to precision agriculture.
Regardless of the type of crop, farming inherently has always been a data driven operation. The methods used to collect and analyze data, however, have been changing fairly rapidly in the past several years. As a result, growers have started to go from clipboards and excel spreadsheets to IoT data collection and business intelligence. Row crops have certainly led the way on automated data collection, given the large number of acres per farm compared to other types of crops. Specialty crops, however, have been catching up in the past couple of years with several new AgTech companies coming online that focus on crops such as wine grapes and tree fruit.
This has significant benefits to a grower, allowing for more focused data collection in areas such as labor cost efficiency or the yield impact of spray applications. More opportunity for farmers though, means more vendors wanting to cash in on data driven farming. This has led to several hundred new companies coming online, talking about how their platform can support better data collection and analysis.
Where to start
Before jumping into the mess of AgTech vendors though, it’s important to step identify the 4 pieces you’re going to need to modernize your data collection and analysis across your agriculture operations. This will help ensure you have a strong foundation from which to build from, if you don’t have these pieces already.
Before setting up all the other pieces of the ecosystem, my advice is to start with data analysis as the first step (even though it’s listed at the end of this post) in order to understand what you want to measure. It’s easy to incur a lot of costs quickly, if you bite off too much at once when it comes to building up a modern day data ecosystem. Having a clear set of KPIs / OKRs / metrics / etc will allow you to tie what you’re measuring to what you’re saving / gaining and get to a return on investment faster. From there, you can add to what you’ve built, and measure more as your use of data grows across the company and more people gain familiarity with the new capabilities.
Part 1) Data Collection
Recently there has been a large surge in sensors for farms that can track everything from soil moisture to pest counts. There’s two things to focus on when considering what tools to use for data collection. The first is what concentration you’re going to need the sensors for accurate data collection. The second is how friendly the sensors are to having data harvested from whatever system they’re feeding their findings into.
Most vendors offer web based solutions for showing you their data, but it’s only really useful if you’re able to extract it into a central place (such as a data warehouse). This is important because you will need to pull together individual readings over a given acre to then compare to an output metric such as yield. As for concentration, without sensors the data is likely already being collected today using manual sampling methods. It’s important, then, to consider the ROI from automated data collection at a sufficient concentration. And of course, always start with just a couple sensors to validate the technology will work for you before scaling.
Part 2) Data Consolidation (or ELT)
Once you have your data collection sorted out, and you’ve begun to measure data in an automated or semi-automated fashion, you’re going to need to get the data into a central spot, such as a data warehouse, in order to put that data to best use. In that case, you’re going to need to pick a vendor to help with this. Though there are ways to do it yourself, it requires hiring 1 or more experienced data engineers / programmers that can build and maintain the system.
For most farming operations, picking an ELT (extract-load-transform) vendor is going to be a better bet, such as FiveTran or Dell Boomi. These tools will allow you to connect to a data source, such as your accounting system or field data collection, and extract / load / transform the data into a central place which then allows you access to all of your raw data across as many systems as you’re able to connect to. This is why, when considering data collection, that you understand what vendors have capabilities such as an API which allows for data extraction.
Part 3) Data Warehousing
The next component that is foundational to being data driven, is having a central location for your data. Depending on the volume of data, you may be able to use a standard database such as MySQL or SQL Server. If the volume is large enough (1 Terabyte or more) you may need to consider a Columnar based database such as Snowflake or Azure Synapse which is designed for large data sets. When you’re storing a lot of data from sensors across your operation, you are likely going to get larger data volumes sooner than later. The key is to start with smaller data sets that are useful right away vs starting big by storing everything and having to support too much at once.
It’s also recommended to host your data in the cloud, and to get as of your ecosystem into the cloud as possible. Though you likely have data hosted locally today, which in some cases is required for business continuity, having a plan to get as much into the cloud as possible has a number of benefits.
Part 4) Data Analysis
Once you are measuring the right information, and then extracting / storing it in a central location, you need to consider what metrics you are going to track. Having 10 or so metrics from which to measure your biggest cost drivers (such as labor or parts) means you can get actionable insights faster vs guessing what might be useful to use all this data on. It’ll also help you understand what data to collect first, vs boiling the ocean and gathering all the data at once. Using a BI, or Business Intelligence tool such as Tableau or PowerBI, will allow you to point to your data warehouse and begin to construct these metrics (or KPIs as they’re often called) in order to begin building automated reporting.
It’s recommended to work with a third-party firm, or experience resource that has worked with these tools in the past to get started. But there is a ton of great resources online on getting started with a BI tool if you want to start getting your hands dirty. You can even start with just an excel or CSV file as a data “source” if you want to get started with BI / reporting prior to having everything else done. Once you have these metrics established, you can look at other capabilities under categories such as data science, AI, or Machine Learning. However, that will require specialized expertise to build solutions using these advanced methods. The good news is these capabilities have been growing over the years, so it’s a lot easier to find expertise with these skills than it used to be a couple years ago.
Though this is a very simplified view of a data ecosystem as there are several complicating factors to consider, this is a good 30,000 foot view of the pieces are essential in all cases of a modern data driven architecture. These should be considered when you’re looking to bring a new AgTech vendor online. Understanding where they fit, and what they’ll be contributing to is critical to ensure you don’t have a bunch of isolated (or “silo’ed”) point solutions that you aren’t able to bring together into a single set of metrics.
The next step is to learn more about each, which I’ll be covering in future blog entries. Of course, if you have any questions, feel free to reach out directly or leave a comment.